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5 This patent application is a continuation-in-part of U.S. Patent Application 

Serial No. 10/095,273 filed on March 12, 2002, the disclosure of which is 
incorporated herein by reference. This patent application is also a continuation- 
in-part of U.S. Patent Application Serial No. 10/162,524 filed on June 5, 2002, the 
disclosure of which is incorporated herein by reference. This patent application 
10 is additionally a continuation-in-part of U.S. Patent Application Serial No. 
10/226,191 filed on August 22, 2002, the disclosure of which is incorporated 
herein by reference. 



TECHNICAL FIELD 

15 [001] Embodiments generally relates to molecular technology, including 

nanotechnology. Embodiments also relate to neural networks, neural computing 
systems and teaching methods thereof. Embodiments also relate to physical 
neural networks implemented utilizing molecular technology, including 
nanotechnology. Embodiments additionally relate to temporal pattern recognition 

20 methods and systems. 

BACKGROUND OF THE INVENTION 

[002] Neural networks are computational systems that permit computers 
to essentially function in a manner analogous to that of the human brain. Neural 
networks do not utilize the traditional digital model of manipulating 0's and 1's. 
25 Instead, neural networks create connections between processing elements, 
which are equivalent to neurons of a human brain. Neural networks are thus 
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based on various electronic circuits that are modeled on human nerve cells (i.e., 
neurons). 

[003] Generally, a neural network is an information-processing network, 
which is inspired by the manner in which a human brain performs a particular 
5 task or function of interest. Computational or artificial neural networks are thus 
inspired by biological neural systems. The elementary building blocks of 
biological neural systems are the neuron, the modifiable connections between 
the neurons, and the topology of the network. 

[004] Biologically inspired artificial neural networks have opened up 
10 new possibilities to apply computation to areas that were previously thought to be 
the exclusive domain of human intelligence. Neural networks learn and 
remember in ways that resemble human processes. Areas that show the 
greatest promise for neural networks, such as pattern classification tasks, speech 
and image recognition, are areas where conventional computers and data- 
15 processing systems have had the greatest difficulty. 

[005] In general, artificial neural networks are systems composed of 
many nonlinear computational elements operating in parallel and arranged in 
patterns reminiscent of biological neural nets. The computational elements, or 
nodes, are connected via variable weights that are typically adapted during use 
20 to improve performance. Thus, in solving a problem, neural net models can 
explore many competing hypothesis simultaneously using massively parallel 
nets composed of many computational elements connected by links with 
variable weights. 

[006] In contrast, with conventional von Neumann computers, an 
25 algorithm must first be developed manually, and a program of instructions 
written and executed sequentially. In some applications, this has proved 
extremely difficult. This makes conventional computers unsuitable for many 
real-time problems for which we have no efficient algorithm. 
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[007] In a neural network, "neuron-like" nodes can output a signal based 
on the sum of their inputs, the output being the result of an activation function. 
In a neural network, there exists a plurality of connections, which are electrically 
coupled among a plurality of neurons. The connections serve as 
5 communication bridges among of a plurality of neurons coupled thereto. A 
network of such neuron-like nodes has the ability to process information in a 
variety of useful ways. By adjusting the connection values between neurons in 
a network, one can match certain inputs with desired outputs. 

[008] One does not program a neural network. Instead, one "teaches" a 
10 neural network by examples. Of course, there are many variations. For 
instance, some networks do not require examples and extract information 
directly from the input data. The two variations are thus called supervised and 
unsupervised learning. Neural networks are currently used in applications such 
as noise filtering, face and voice recognition and pattern recognition. Neural 
15 networks can thus be utilized as an advanced technique for processing 
information. 

[009] Neural networks that have been developed to date are largely 
software-based. A true neural network (e.g., the human brain) is massively 
parallel (and therefore very fast computationally) and very adaptable. For 
20 example, half of a human brain can suffer a lesion early in its development and 
not seriously affect its performance. Software simulations are slow because 
during the learning phase a standard computer must serially calculate 
connection strengths. When the networks get larger (and therefore more 
powerful and useful), the computational time becomes enormous. 

25 [0010] For example, networks with 10,000 connections can easily 

overwhelm a computer. In comparison, the human brain has about 100 billion 
neurons, each of which can be connected to about 5,000 other neurons. On the 
other hand, if a network is trained to perform a specific task, perhaps taking 
many days or months to train, the final useful result can be built or "downloaded" 

30 onto a piece of hardware and also mass-produced. Because most problems 
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requiring complex pattern recognition are highly specific, networks are task- 
specific. Thus, users usually provide their own, task-specific training data. 



[0011] A number of software simulations of neural networks have been 
developed. Because software simulations are performed on conventional 
5 sequential computers, however, they do not take advantage of the inherent 
parallelism of neural network architectures. Consequently, they are relatively 
slow. One frequently used measurement of the speed of a neural network 
processor is the number of interconnections it can perform per second. 

[0012] For example, the fastest software simulations available can 
10 perform up to approximately 18 million interconnects per second. Such speeds, 
however, currently require expensive super computers to achieve. Even so, 
approximately 18 million interconnects per second is still too slow to perform 
many classes of pattern classification tasks in real time. These include radar 
target classifications, sonar target classification, automatic speaker identification, 
15 automatic speech recognition, electro-cardiogram analysis, etc. 

[0013] The implementation of neural network systems has lagged 
somewhat behind their theoretical potential due to the difficulties in building 
neural network hardware. This is primarily because of the large numbers of 
neurons and weighted connections required. The emulation of even of the 
20 simplest biological nervous systems would require neurons and connections 
numbering in the millions and/or billions. 

[0014] Due to the difficulties in constructing such highly interconnected 
processors, currently available neural network hardware systems have not 
approached this level of complexity. Another disadvantage of hardware systems 
25 is that they typically are often custom designed and configured to implement one 
particular neural network architecture and are not easily, if at all, reconfigurable 
in implementing different architectures. A.true physical neural network chip, with 
the learning abilities and connectivity of a biological network, has not yet been 
designed and successfully implemented. 
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[0015] The problem with a pure hardware implementation of a neural 
network utilizing existing technology is the inability to physically form a great 
number of connections and neurons. On-chip learning can exist, but the size of 
the network is limited by digital processing methods and associated electronic 
5 circuitry. One of the difficulties in creating true physical neural networks lies in 
the highly complex manner in which a physical neural network must be designed 
and constructed. The present inventor believes that solutions to creating a true 
physical and artificial neural network lie in the use of nanotechnology and the 
implementation of a novel form of variable connections. 

10 [0016] The term "Nanotechnology" generally refers to nanometer-scale 

manufacturing processes, materials and devices, as associated with, for 
example, nanometer-scale lithography and nanometer-scale information storage. 
Nanometer-scale components find utility in a wide variety of fields, particularly in 
the fabrication of microelectrical and microelectromechanical systems (commonly 

15 referred to as "MEMS"). Microelectrical nano-sized components include 
transistors, resistors, capacitors and other nano-integrated circuit components. 
MEMS devices include, for example, micro-sensors, micro-actuators, micro- 
instruments, micro-optics, and the like. 

[0017] In general, nanotechnology presents a solution to the problems 
20 faced in the rapid pace of computer chip design in recent years. According to 
Moore's law, the number of switches that can be produced on a computer chip 
has doubled every 18 months. Chips now can hold millions of transistors. It is, 
however, becoming increasingly difficult to increase the number of elements on a 
chip utilizing existing technologies. At the present rate, in the next few years the 
25 theoretical limit of silicon-based chips will have been attained. Because the 
number of elements and components that can be manufactured on a chip 
determines the data storage and processing capabilities of microchips, new 
technologies are required for the development of higher performance chips. 

[0018] Present chip technology is also limited in cases where wires must 
30 be crossed on a chip. For the most part, the design of a computer chip is limited 
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to two dimensions. Each time a circuit is forced to cross another circuit, another 
layer must be added to the chip. This increases the cost and decreases the 
speed of the resulting chip. A number of alternatives to standard silicon based 
complementary metal oxide semiconductor ("CMOS") devices have been 
5 proposed. The common goal is to produce logic devices on a nanometer scale. 
Such dimensions are more commonly associated with molecules than integrated 
circuits. 

[0019] The issue of interconnects in neural network hardware poses a 
serious problem. Because of the massive interconnectivity, a neural network 
10 constructed with standard integrated electronic methods can never reach the 
desired neuron and synapse density, simply because the interconnections 
overwhelm the largely 2-diminsional chip. It can thus be appreciated that almost 
any sort of 3-diminsional connectivity, no matter how simple, could offer 
tremendous benefits. 

15 [0020] Integrated circuits and electrical components thereof, which can 

be produced at a molecular and nanometer scale, include devices such as 
carbon nanotubes and nanowires, which essentially are nanoscale conductors 
("nanoconductors"). Nanoconductors are tiny conductive tubes (i.e., hollow) or 
wires (i.e., solid) with a very small size scale (e.g., 0.7 to 300 nanometers in 

20 diameter and up to 1mm in length). Their structure and fabrication have been 
widely reported and are well known in the art. Carbon nanotubes, for example, 
exhibit a unique atomic arrangement, and possess useful physical properties 
such as one-dimensional electrical behavior, quantum conductance, and ballistic 
electron transport. 

25 [0021] Carbon nanotubes are among the smallest dimensioned nanotube 

materials with a generally high aspect ratio and small diameter. High-quality 
single-walled carbon nanotubes can be grown as randomly oriented, needle-like 
or spaghetti-like tangled tubules. They can be grown by a number of fabrication 
methods, including chemical vapor deposition (CVD), laser ablation or electric 

30 arc growth. 
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[0022] Carbon nanotubes can be grown on a substrate by catalytic 
decomposition of hydrocarbon containing precursors such as ethylene, methane, 
or benzene. Nucleation layers, such as thin coatings of Ni, Co, or Fe are often 
intentionally added onto the substrate surface in order to nucleate a multiplicity of 
5 isolated nanotubes. Carbon nanotubes can also be nucleated and grown on a 
substrate without a metal nucleating layer by using a precursor including one or 
more of these metal atoms. Semiconductor nanowires can be grown on 
substrates by similar processes. 

[0023] Attempts have been made to construct electronic devices utilizing 
10 nano-sized electrical devices and components. For example, a memory device 
that can be constructed from crossbar arrays of nanowires sandwiching 
molecules that act as on/off switches. Such a device can be formed from a 
plurality of nanometer-scale devices, each device comprising a junction formed 
by a pair of crossed wires where a single wire crosses another and at least one 
15 connector species connects the pair of crossed wires in the junction. The 
connector species comprises a bi-stable molecular switch. The junction forms 
either a resistor or a diode or an asymmetric non-linear resistor. The junction has 
a state that is capable of being altered by application of a first voltage and 
sensed by the application of a second, non-destructive voltage. A series of 
20 related patents attempts to cover everything from molecular logic to how to 
chemically assemble these devices. 

[0024] Such a molecular crossbar device has two general applications. 
The notion of transistors built from nanotubes and relying on nanotube properties 
is being pursued. Second, two wires can be selectively brought to a certain 

25 voltage and the resulting electrostatic force attracts them. When they touch, the 
Van der Walls force keeps them in contact with each other and a "bit" is stored. 
The connections in this apparatus can therefore be utilized for a standard (i.e., 
binary and serial) computer. The inventors of such a device thus desire to coax a 
nanoconductor into a binary storage media or a transistor. As it turns out, such a 

30 device is easier to utilize as a storage device. 
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[0025] A need exists for a physical neural network, which can be 
implemented in the context of a semiconductor integrated circuit (i.e., a computer 
chip). Such a device, which can be referred to as a "physical neural network 
chip" or a "synapse chip" is thus disclosed herein. Such a device, if successfully 
5 implemented would be well suited for use with neural networks. 

[0026] Researchers in the neuro-biological fields have attempted to 
develop a computationally efficient algorithm that can emulate a biologically 
realistic neural network. Specifically, researchers have attempted to develop 
method and/or systems, which would allow the efficient calculation of Spike- 
10 Timing Dependent-Plasticity (STDP), while also permitting fully interconnected 
pulsed networks. In STDP, timing between pre- and post-synaptic events can 
cause a net potentiation (LTP) or a net depression (LTD) of synapses. 

[0027] A background for STPD is explained in "Mechanisms and 
Significance of Spike-Timing Dependent Plasticity" by Uma R. Karmarkar, et al., 

15 Biol. Cybern. 87, 373-382 (2002). STPD is also explained generally in "A Model 
of Spike-Timing Dependent Plasticity: One or Two Coincidence Detectors?" by 
Uma R. Karmarkar, et al., J. Neurophysiol, 88: 507-513, 2002. An additional 
reference regarding STDP is "Spike-Based Learning Rules and Stabilization of 
Persistent Neural Activity," by Xiaohui Xie, et al., Dept. of Brain & Cog, Sci, MIT, 

20 Cambridge, MA. A further reference regarding STDP and issues faced thereof is 
described in "Dendritic Spikes as a Mechanism for Cooperative Long-Term 
Potentiation," Nace L. Golding, et al., Nature, Vol. 418, 18 July 2002, pp. 326- 
330. The aforementioned references are indicated herein generally for 
background and edification purposes only. 

25 [0028] Another type of neural network, which has been proposed, is 

known as a liquid state machine (LMS). A non-limiting and non-essential 
example of an LMS is disclosed in "Computational Models for Generic Cortical 
Microcircuits" by Wolfgang Maass, et al., Institute for Theoretical Computer 
Science, Technische Universitaet Graz, Graz, Austria, June 10, 1003. Note that 

30 the aforementioned Maass et al reference is referred to herein for general 
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edification and background purposes only. It is believed that liquid state 
machines have not been implemented in the context of physical neural networks 
configured based on nanotechnology. A need thus exists for such devices, 
including methods and systems thereof. 
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BRIEF SUMMARY OF THE INVENTION 



[0029] The following summary of the invention is provided to facilitate an 
understanding of some of the innovative features unique to the present invention, 
5 and is not intended to be a full description. A full appreciation of the various 
aspects of the invention can be gained by taking the entire specification, claims, 
drawings, and abstract as a whole. 

[0030] It is, therefore, one aspect of the present invention to provide for a 
physical neural network, including an adaptive neural network, which can be 
10 formed and implemented utilizing nanotechnology. 

[0031] It is still another aspect of the present invention to provide a 
physical neural network, which can be formed from a plurality of interconnected 
molecular connections, such as, for example, molecules, nanoconnections, 
and/or nanoconnectors. 

15 [0032] It is yet a further aspect of the present invention to provide a 

physical neural network, which can be formed from a plurality of molecules, 
including molecular conducting structures. 

[0033] It a further aspect of the present invention to provide a physical 
neural network based on nanoconductors, such as, for example, nanowires 
20 and/or nanotubes. 

[0034] It is also an aspect of the present invention to provide physical, 
large scale, arbitrarily connected, pulsed and synaptically modifiable neural 
networks. 

[0035] It is still an additional aspect of the present invention to provide a 
25 physical neural network, which can be implemented physically in the form of a 
chip structure. 
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[0036] The above and other aspects can be achieved as will now be 
described. A physical neural network is disclosed herein, which comprises a 
liquid state machine. The physical neural network is configured from molecular 
connections located within a dielectric solvent between pre-synaptic and post- 
5 synaptic electrodes thereof, such that the molecular connections are 
strengthened or weakened according to an application of an electric field or a 
frequency thereof to provide physical neural network connections thereof. A 
supervised learning mechanism is associated with the liquid state machine, 
whereby connections strengths of the molecular connections are determined by 
10 pre-synaptic and post-synaptic activity respectively associated with the pre- 
synaptic and post-synaptic electrodes, wherein the liquid state machine 
comprises a dynamic fading memory mechanism. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



[0037] The accompanying figures, in which like reference numerals refer 
to identical or functionally-similar elements throughout the separate views and 
5 which are incorporated in and form part of the specification, further illustrate the 
present invention and, together with the detailed description of the invention, 
serve to explain the principles of the present invention. 

[0038] FIG. 1 illustrates a graph illustrating a typical activation function; 

[0039] FIG. 2 illustrates a schematic diagram illustrating a diode 
10 configuration as a neuron, in accordance with one embodiment of the present 
invention; 

[0040] FIG. 3 illustrates a block diagram illustrating a network of 
nanoconnections formed between two electrodes, in accordance with one 
embodiment of the present invention; 

15 [0041] FIG. 4 illustrates a block diagram illustrating a plurality of 

connections between inputs and outputs of a physical neural network, in 
accordance with one embodiment of the present invention; 

[0042] FIG. 5 illustrates a schematic diagram of a physical neural 
network that can be created without disturbances, in accordance with one 
20 embodiment of the present invention; 

[0043] FIG. 6 illustrates a schematic diagram illustrating an example of a 
physical neural network that can be implemented in accordance with an 
alternative embodiment of the present invention; 

[0044] FIG. 7 illustrates a schematic diagram illustrating an example of a 
25 physical neural network that can be implemented in accordance with an 
alternative embodiment of the present invention; 
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[0045] FIG. 8 illustrates a schematic diagram of a chip layout for a 
connection network that may be implemented in accordance with an alternative 
embodiment of the present invention; 



[0046] FIG. 9 illustrates a flow chart of operations illustrating operational 
5 steps that can be followed to construct a connection network, in accordance with 
one embodiment of the present invention; 

[0047] FIG. 10 illustrates a flow chart of operations illustrating operational 
steps that can be utilized to strengthen nanoconductors within a connection gap, 
in accordance with one embodiment of the present invention; 

10 [0048] FIG. 11 illustrates a schematic diagram of a circuit illustrating 

temporal summation within a neuron, in accordance with one embodiment of the 
present invention; 

[0049] FIG. 12 illustrates a block diagram illustrating a pattern recognition 
system, which can be implemented with a physical neural network device, in 
1 5 accordance with an alternative embodiment of the present invention; 

[0050] FIG. 13 illustrates a schematic diagram of a 2-input, 1 -output, 2- 
layer inhibitory physical neural network, which can be implemented in 
accordance with one embodiment of the present invention; 

[0051] FIG. 14 illustrates a pictorial diagram of a perspective view of a 
20 synapse array, which can be implemented in accordance with one embodiment 
of the present invention; 

[0052] FIG. 15 illustrates a pictorial diagram of a perspective view of an 
alternative chip structure with parallel conductors on output, which can be 
implemented in accordance with an alternative embodiment of the present 
25 invention; 
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[0053] FIG. 16 illustrates a perspective view of a system that includes a 
connection formation, in accordance with a preferred or alternative embodiment 
of the present invention; 

[0054] FIG. 17 illustrates a system illustrating the use of system of FIG. 
5 16 in the context of a synapse chip and neural network configuration thereof; 

[0055] FIG. 18 illustrates a schematic diagram of electrode widths 
encoding specific synapses resistances, in accordance with an alternative 
embodiment of the present invention; 

[0056] FIG. 19 illustrates a schematic diagram of one example of an 
10 adaptive integration network comprising six interconnected processing elements, 
in accordance with an alternative embodiment of the present invention; 

[0057] FIG. 20 illustrates a schematic diagram of an adaptive integration 
network with mutually interacting loops that share a connection, in accordance 
with one embodiment of the present invention; 

15 [0058] FIG. 22 illustrates a graph illustrating exemplary connection 

weight strengthening and connection weight weakening curves in accordance 
with one embodiment of the present invention. 

[0059] FIG. 23 illustrates a flowchart illustrating the operation of adaptive 
learning in accordance with one embodiment of the present invention; 

20 [0060] FIGS. 24 and 25 illustrates before and after schematic drawings of 

an exemplary adaptive integration network for illustrating how an active pathway 
is dislodged in accordance with one embodiment of the present invention; 

[0061] FIG. 26 illustrates a flow chart of operations depicting logical 
operational steps for modifying a synapse of a physical neural network, in 
25 accordance with an alternative embodiment of the present invention; 

[0062] FIG. 27 illustrates a flow chart of operations illustrating logical 
operational steps for strengthening one or more nanoconnections of a connection 
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network of a physical neural network by an increase in frequency, in accordance 
with an alternative embodiment of the present invention; 

[0063] FIGS. 28 and 29 illustrate respective graphs of varying Spike- 
Time Dependent Plasticity (STPD) models; 

5 [0064] FIG. 30 illustrates a high-level block diagram of a neuron, 

including a dendrite and axon thereof, which can be implemented in accordance 
with an embodiment of the present invention; 

[0065] FIG. 31 illustrates a block diagram of an excitatory axonal pulse 
and an excitatory dendritic pulse, which can be implemented in accordance with 
1 0 an embodiment of the present invention; 

[0066] FIG. 32 illustrates a high-level diagram of two neurons, which can 
be implemented in accordance with an embodiment of the present invention; 

[0067] FIG. 33 illustrates a series of pulses, which can be implemented in 
accordance with an embodiment of the present invention; 

15 [0068] FIG. 34 illustrates a graph, which can be generated in accordance 

with an embodiment of the present invention; 

[0069] FIG. 35 illustrates alternative pulses which can be implemented in 
accordance with an embodiment of the present invention; 

[0070] FIG. 36 illustrates a graph, which can be generated in accordance 
20 with an embodiment of the present invention; 

[0071] FIG. 37 illustrates a high-level block diagram illustrating a system 
comprising a network of nanoconnections formed between one or more 
respective input and output electrodes, in accordance with an alternative 
embodiment of the present invention; 

25 [0072] FIG. 38 illustrates a high-level block diagram illustrating a system 

3800 comprising a network of nanoconnections formed between one or more 
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respective input and output electrodes, in accordance with an alternative 
embodiment of the present invention; 



[0073] FIG. 39 illustrates a block diagram of a physical neural network 
3900, which can be implemented in accordance with an alternative embodiment 
5 of the present invention; 

[0074] FIG. 40 illustrates a block diagram of a system that includes the 
physical neural network of FIG. 39 in association with a plurality of perceptrons, 
which can be implemented in accordance with an alternative embodiment of the 
present invention. 

10 [0075] FIG. 41 illustrates a schematic diagram of a physical neural 

network, which can be implemented in accordance with an alternative 
embodiment of the present invention; 

[0076] FIG. 42 illustrates a schematic diagram of a system that includes 
the physical neural network of FIG. 41 and perceptrons thereof, which can be 
15 implemented in accordance with an alternative embodiment of the present 
invention; and 

[0077] FIG. 43 illustrates a schematic diagram of a perceptron, which can 
be implemented in accordance with an alternative embodiment of the present 
invention. 

20 
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DETAILED DESCRIPTION OF THE INVENTION 



[0078] The particular values and configurations discussed in these non- 
limiting examples can be varied and are cited merely to illustrate an embodiment 
5 of the present invention and are not intended to limit the scope of the invention. 

[0079] The physical neural network described and disclosed herein is 
different from prior art forms of neural networks in that the disclosed physical 
neural network does not require computer calculations for training, nor is its 
architecture based on any current neural network hardware device. The physical 
10 neural network described herein is generally fast and adaptable, no matter how 
large such a physical neural network becomes. 

[0080] The physical neural network described herein with respect to one 
or more embodiments can be referred to generically as a Knowm m . The terms 
"physical neural network" and "Knowm" can thus be utilized interchangeably to 
15 refer to the same device, network, or structure. The term "Knowm" can also refer 
to a semiconductor implementation, such as a physical neural network chip 
and/or synapse chip. Note that the terms "physical neural network chip" and 
"synapse chip" can also be utilized herein to refer generally to the same or 
analogous type of Knowm™ device. 

20 [0081] Network orders of magnitude larger than current VSLI neural 

networks can now be built. One consideration for a Knowm™ is that it must be 
large enough for its inherent parallelism to shine through. Because the 
connection strengths of such a physical neural network are dependant on the 
physical movement of nanoconnections thereof, the rate at which a small 

25 network can learn is generally very small and a comparable network simulation 
on a standard computer can be very fast. On the other hand, as the size of the 
network increases, the time to train the device does not change. Thus, even if the 
network takes a full second to change a connection value a small amount, if it 
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does the same to a billion connections simultaneously, then its parallel nature 
begins to express itself. 

[0082] A physical neural network (i.e., a Knowm™) must have two 
components to function properly. First, the physical neural network must have 
5 one or more neuron-like nodes that sum a signal and output a signal based on 
the amount of input signal received. Such a neuron-like node is generally non- 
linear in output. In other words, there should be a certain threshold for input 
signals, below which nothing is output and above which a constant or nearly 
constant output is generated or allowed to pass. This is considered the basic 
10 building block of all neural networks, and can be accomplished by an activation 
function. The second requirement of a physical neural network is the inclusion of 
a connection network composed of a plurality of interconnected electrodes (i.e., 
nanoconnections). Such a connection network is described in greater detail 
herein. 

[0083] FIG. 1 illustrates a graph 100 illustrating a typical activation 
function that can be implemented in accordance with the physical neural network 
of the present invention. Note that the activation function need not be non-linear, 
although non-linearity is generally desired for learning complicated input-output 
relationships. The activation function depicted in FIG. 1 comprises a linear 
function, and is shown as such for general edification and illustrative purposes 
only. As explained previously, an activation function may also be non-linear. 

[0084] As illustrated in FIG. 1, graph 100 includes a horizontal axis 104 
representing a sum of inputs, and a vertical axis 102 representing output values. 
A graphical line 106 indicates threshold values along a range of inputs from 
approximately -10 to +10 and a range of output values from approximately 0 to 1 . 
As more neural networks (i.e., active inputs) are established, the overall output 
as indicated at line 105 climbs until the saturation level indicated by line 106 is 
attained. If a connection is not utilized, then the level of output (i.e., connection 
strength) begins to fade until it is revived. This phenomenon is analogous to short 
term memory loss of a human brain. Note that graph 100 is presented for 
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15 



20 



25 



30 



generally illustrative and edification purposes only and is not considered a 
limiting feature of the present invention. 

[0085] In a Knowm™, the neuron-like node can be configured as a 
standard diode-based circuit, the diode being the most basic semiconductor 
5 electrical component, and the signal it sums can be a voltage. An example of 
such an arrangement of circuitry is illustrated in FIG. 2, which generally illustrates 
a schematic diagram illustrating a diode-based configuration as a neuron 200, in 
accordance with an embodiment of the present invention. The use of such a 
diode-based configuration is not considered a limiting feature of the present 
10 invention, but merely represents one potential arrangement in which the present 
invention can be implemented. 

[0086] Although a diode may not necessarily be utilized, its current 
versus voltage characteristics are non-linear when used with associated resistors 
and similar to the relationship depicted in FIG. 1 . The use of a diode as a neuron 

15 is thus not considered a limiting feature of the present invention, but is only 
referenced herein with respect to one potential embodiment of the present 
invention. The use of a diode and associated resistors with respect to an 
embodiment simply represents one possible "neuron" implementation. Such a 
configuration can be said to comprise an artificial neuron. It is anticipated that 

20 other devices and components can be utilized instead of a diode to construct a 
physical neural network and a neuron-like node (i.e., artificial neuron), as 
indicated herein. 

[0087] Thus, neuron 200 comprises a neuron-like node that may include 
a diode 206, which is labeled Di, and a resistor 204, which is labeled R 2 . Resistor 

25 204 is connected to a ground 210 and an input 205 of diode 206. Additionally, a 
resistor 202, which is represented as a block and labeled can be connected 
to input 205 of diode 206. Block 202 includes an input 212, which comprises an 
input to neuron 200. A resistor 208, which is labeled R3, is also connected to an 
output 214 of diode 206. Additionally, resistor 208 is coupled to ground 210. 

30 Diode 206 in a physical neural network is analogous to a neuron of a human 
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brain, while an associated connection formed thereof, as explained in greater 
detail herein, is analogous to a synapse of a human brain. 

[0088] As depicted in FIG. 2, the output 214 is determined by the 
connection strength of Ri (i.e., resistor 202). If the strength of R/s connection 
5 increases (i.e., the resistance decreases), then the output voltage at output 214 
also increases. Because diode 206 conducts essentially no current until its 
threshold voltage (e.g., approximately .6V for silicon) is attained, the output 
voltage will remain at zero until Ri conducts enough current to raise the pre- 
diode voltage to approximately .6V. After .6V has been achieved, the output 
10 voltage at output 214 will increase linearly. Simply adding extra diodes in series 
or utilizing different diode types may increase the threshold voltage. 

[0089] An amplifier may also replace diode 206 so that the output voltage 
immediately saturates at a reference threshold voltage, thus resembling a step 
function. R3 (i.e., resistor 208) functions generally as a bias for diode 206 (i.e., 
15 D1) and should generally be about 10 times larger than resistor 204 (i.e., R 2 ). In 
the circuit configuration illustrated in FIG. 2, Ri can actually be configured as a 
network of connections composed of many inter-connected conducting 
nanowires (i.e., see FIG. 3). As explained previously, such connections are 
analogous to the synapses of a human brain. 

20 [0090] FIG. 3 illustrates a block diagram illustrating a system 300 that 

includes, but is not limited to, a network of nanoconnections 304 formed between 
two or more electrodes, in accordance with one embodiment of the present 
invention. Nanoconnections 304 (e.g., nanoconductors) depicted in FIG. 3 can be 
located between input 302 and output 306. The network of nanoconnections 

25 depicted in FIG. 3 can be implemented as a network of molecules, including, for 
example, nanoconductors. Examples of nanoconductors include devices such 
as, for example, nanowires, nanotubes, and nanoparticles. 

[0091] Nanoconnections 304, which are analogous to biological 
synapses, can be composed of electrical conducting material (i.e., 
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nanoconductors). Nanoconductors can be provided in a variety of shapes and 
sizes without departing from the teachings herein. A nanoconductor can also be 
implemented as, for example, a molecule or groups of molecules. A 
nanoconductor can also be implemented as, for example, DNA. Studies have 
5 shown that DNA has special electrical properties which can function as 
essentially a tiny electrical wire. This recent discovery opens up a possible route 
to new applications in the electronics industry and particularly with respect to the 
physical neural network disclosed herein. 

[0092] Carbon particles (e.g., granules or bearings) can also be utilized 
10 for developing nanoconnections. The nanoconductors utilized to form a 
connection network can be formed as a plurality of nanoparticles. Thus, 
nanoconductors that are utilized to form a physical neural network (i.e., 
Knowm™) can be formed from such nanoparticles. Note that as utilized herein, 
the term "nanoparticle" can be utilized interchangeably with the term 
15 "nanoconductor." The term "nanoparticle" can refer simply to a particular type of 
nanoconductors, such as, for example, a carbon nanoparticle, or another type of 
nanoconductors, such as, for example, a carbon nanotube or carbon nanowire. 
Devices that conduct electricity and have dimensions on the order of nanometers 
can be referred to as nanoconductors. 

20 [0093] A connection network as disclosed herein can be composed from 

a variety of different types of nanoconductors. For example, a connection 
network can be formed from a plurality of nanoconductors, including nanowires, 
nanotubes and/or nanoparticles. Note that such nanowires, nanotubes and/or 
nanoparticles, along with other types of nanoconductors can be formed from 

25 materials such as carbon or silicon. For example, carbon nanotubes may 
comprise a type of nanotube that can be utilized in accordance with the present 
invention. 

[0094] As illustrated in FIG. 3, nanoconnections 304 comprise a plurality 
of interconnected nanoconnections, which can be referred to generally as a 
30 "connection network." An individual nanoconnection may constitute a 
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nanoconductor such as, for example, a nanowire, a nanotube, nanoparticles(s), 
or any other nanoconducting structures. Nanoconnections 304 may comprise a 
plurality of interconnected nanotubes and/or a plurality of interconnected 
nanowires. Similarly, nanoconnections 304 can be formed from a plurality of 
5 interconnected nanoparticles. 

[0095] A connection network is thus not one connection between two 
electrodes, but a plurality of connections between input electrodes and output 
electrodes. Nanotubes, nanowires, nanoparticles and/or other nanoconducting 
structures can be utilized, of course, to construct nanoconnections 304 between 
10 input 302 and input 306. Although a single input 302 and a single input 306 is 
depicted in FIG. 3, it can be appreciated that a plurality of inputs and a plurality of 
outputs can be implemented in accordance with the present invention, rather 
than simply a single input 302 or a single output 306. 

[0096] FIG. 4 illustrates a block diagram illustrating a plurality of 
15 connections 414 between inputs 404, 406, 408, 410, 412 and outputs 416 and 
418 of a physical neural network, in accordance with one embodiment of the 
present invention. Inputs 404, 406, 408, 410, and 412 provide input signals to 
connections 414. Output signals are then generated from connections 414 via 
outputs 416 and 418. A connection network can thus be configured from the 
20 plurality of connections 414. Such a connection network is generally associated 
with one or more neuron-like nodes. 

[0097] The connection network also comprises a plurality of 
interconnected nanoconnections, wherein each nanoconnection thereof is 
strengthened or weakened according to an application of an electric field. A 

25 connection network is not possible if built in one layer because the presence of 
one connection can alter the electric field so that other connections between 
adjacent electrodes could not be formed. Instead, such a connection network can 
be built in layers, so that each connection thereof can be formed without being 
influenced by field disturbances resulting from other connections. This can be 

30 seen in FIG. 5. 
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[0098] FIG. 5 illustrates a schematic diagram of a physical neural 
network 500 that can be created without disturbances, in accordance with one 
embodiment of the present invention. Physical neural network 500 is composed 
of a first layer 558 and a second layer 560. A plurality of inputs 502, 504, 506, 
5 508, and 510 can be respectively provided to layers 558 and 560 respectively via 
a plurality of input lines 512, 514, 516, 518, and 520 and a plurality of input lines 
522, 524, 526, 528, and 530. Input lines 512, 514, 516, 518, and 520 are further 
coupled to input lines 532, 534, 536, 538, and 540 such that each line 532, 534, 
536, 538, and 540 is respectively coupled to nanoconnections 572, 574, 576, 
10 578, and 580. Thus, input line 532 can be connected to nanconnections 572. 
Input line 534 can be connected to nanoconnections 574, and input line 536 can 
be connected to nanoconnections 576. Similarly, input line 538 can be 
connected to nanconnections 578, and input line 540 is generally connected to 
nanoconnections 580. 

15 [0099] Nanconnections 572, 574, 576, 578, and 580 may comprise 

nanoconductors such as, for example, nanotubes and/or nanowires. 
Nanoconnections 572, 574, 576, 578, and 580 thus comprise one or more 
nanoconductors. Additionally, input lines 522, 524, 526, 528, and 530 are 
respectively coupled to a plurality of input lines 542, 544, 546, 548 and 550, 

20 which are in turn each respectively coupled to nanoconnections 582, 584, 586, 
588, and 590. 

[00100] Thus, for example, input line 542 is connected to nanoconnections 
582, while input line 544 is connected to nanoconnections 584. Similarly, input 
line 546 is connected to nanoconnections 586 and input line 548 is connected to 
25 nanoconnections 588. Additionally, input line 550 is connected to 
nanconnections 590. Box 556 and 554 generally represent simply the output 
and are thus illustrated connected to outputs 562 and 568. In other words, 
outputs 556 and 554 respectively comprise outputs 562 and 568. The 
aforementioned input lines and associated components thereof actually comprise 
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physical electronic components, including conducting input and output lines and 
physical nanoconnections, such as nanotubes and/or nanowires. 



[00101] Thus, the number of layers 558 and 560 equals the number of 
desired outputs 562 and 568 from physical neural network 500. In the previous 
5 two figures, every input was potentially connected to every output, but many 
other configurations are possible. The connection network can be made of any 
electrically conducting material, although the practicality of the application 
requires that they be very small so that they will align with a practical voltage. 
Carbon nanotubes or any conductive nanowire can be implemented in 
10 accordance with the physical neural network described herein. 

[00102] Such components can thus form connections between electrodes 
by the presence of an electric field. For example, the orientation and purification 
of carbon nanotubes can be implemented using ac electrophoresis in isopropyl 
alcohol. Additionally, for example, an electric-field assisted assembly technique 

15 maybe utilized to position individual nanowires suspended in an electric medium 
between two electrodes defined lithographically on a Si0 2 substrate. It is also 
possible, for example, to fabricate deterministic wiring networks from single- 
walled carbon nanotubes (SWNTs). Also, an electric-field-directed assembly can 
be utilized to create highly integrated device arrays from nanowire building 

20 blocks. 

[00103] The only general requirements for the conducting material utilized 
to configure the nanoconductors are that such conducting material must conduct 
electricity, and a dipole should preferably be induced in the material when in the 
presence of an electric field. Alternatively, the nanoconductors utilized in 
25 association with the physical neural network described herein can be configured 
to include a permanent dipole that is produced by a chemical means, rather than 
a dipole that is induced by an electric field. A connection network could also be 
configured from other conductive particles that are developed or found useful in 
i the nanotechnology arts. For example, carbon particles (e.g., carbon "dust") may 
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also be used as nanoconductors in place of nanowires or nanotubes. Such 
particles may include bearings or granule-like particles. 

[00104] A connection network can be constructed as follows. Initially, a 
voltage can be applied across a gap that is filled with a mixture of nanowires and 
5 a "solvent". This mixture can be composed of a variety of materials or 
substances. The only general requirement in constructing such a connection 
network is that the conducting wires should be suspended in the solvent and/or 
dissolved or in a suspension, but free to move about. Additionally, the electrical 
conductance of the substance should generally be less than the electrical 
10 conductance of the suspended conducting nanowire(s) and/or other 
nanoparticle(s). The viscosity of the substance should not be too much so that 
the conducting nanowire(s) and/or other nanoparticles(s) cannot move when an 
electric field is applied. 

[00105] The goal for such a connection network is to develop a network of 
15 connections of just the "right" values so as to satisfy particular signal-processing 
requirements, which is precisely how a neural network functions. Applying a 
voltage across a space occupied by the aforementioned mixture can form a 
connection network. To create a connection network, input terminals can be 
selectively raised to a positive voltage, while the output terminals can be 
20 selectively grounded. 

[00106] Alternatively, an electric field, either AC or DC, can be applied 
across the terminals. Such an electric field can be, for example, a sinusoidal, 
square or a saw-tooth waveform. Thus, connections can gradually form between 
the inputs and outputs. The important requirement that makes a physical neural 
25 network functional as a neural network in accordance with an embodiment of the 
present invention is that the longer this electric field is applied across the 
connection gap, and/or the greater the frequency or amplitude of the field, the 
more nanotubes and/or nanowires and/or nanoparticles align and the stronger 
the connections thereof become. 
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[00107] The connections can either be initially formed and possess 
random resistances or no connections may be formed at all. By initially forming 
random connections, it might be possible to teach the desired relationships 
faster, because it is not necessary for the base connections to be constructed 
5 from scratch. Depending on the rate of connection decay, having initial random 
connections could prove faster, although not necessarily. The connection 
network can adapt itself to the requirements of a given situation regardless of the 
initial state of the connections. 

[00108] The resistance of a connection can be maintained or lowered by 
10 selective activations of the connection. In other words, an electric field can be 
applied perpendicular to the direction of connection formation by perpendicular 
electrodes. Alternately, both the input and output electrodes could be given the 
same sinusoidal, alternating signal, which would create pulses of electrostatic 
repulsion in the connection region. The temperature of the solution can also be 
15 controlled so that the rate that connection degradation can be controlled. 

[00109] The nanoconnections may or may not be arranged in an orderly 
array pattern between the input and output electrodes. The nanoconnections 
(e.g., nanotubes, nanowires, etc) of a physical neural network do not have to 
order themselves into neatly formed arrays. They simply float in the solution, or 

20 lie at the bottom of the gap, and more or less line up in the presence an electric 
field. Precise patterns are thus not necessary. In fact, neat and precise patterns 
may not be desired. Rather, precise patterns could be a drawback rather than an 
advantage. In fact, it may be desirable that the connections themselves function 
as poor conductors, so that variable connections are formed thereof, overcoming 

25 simply an "on" and "off structure, which is commonly associated with binary and 
serial networks and structures thereof. 

[00110] Although it can be seen that nanoparticles aligned in a dielectric 
solution offer a unique solution to emulated modifiable, variable connections 
within an electronic implementation of a neural network, it is not yet obvious how 
30 one would provide feedback that would train the connections. A training 
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mechanism may be implemented in many different forms. Basically, the 
connections in a connection network must be able to change in accordance with 
the feedback provided. In other words, the very general notion of connections 
being strengthened or connections being weakened in a physical system is the 
5 essence of a physical neural network (i.e., a Knowm™ physical neural network). 

[001 11] Thus, it can be appreciated that the training of such a physical 
neural network may not require a "CPU" to calculate connection values thereof. 
The Knowm™ physical neural network, including artificial synapses thereof, can 
adapt itself. Complicated neural network solutions could be implemented very 

10 rapidly "on the fly", much like a human brain adapts as it performs. It is 
anticipated that various learning mechanisms can be implemented in accordance 
with preferred or alternative embodiments of the present invention. Two such 
learning mechanisms are generally discussed herein. First, a feedback 
mechanism is described that leads to the training of a multi-layer, feed-forward 

15 network. Second, a feedback mechanism is generally discussed, which can 
result in Hebbian synapse modification within recurrent, highly interconnected 
networks. 

[001 12] The physical neural network disclosed herein thus has a number 
of broad applications. The core concept of a Knowm™ physical neural network, 
20 however, is basic. The very basic idea that the connection values between 
electrode junctions by nanoconductors can be used in a neural network devise is 
all that required to develop an enormous number of possible configurations and 
applications thereof. 

[00113] An important feature of a physical neural network is the ability to 
25 form negative connections. This is an important feature that makes possible 
inhibitory effects useful in data processing. The basic idea is that the presence of 
one input can inhibit the effect of another input. In artificial neural networks as 
they currently exist, this is accomplished by multiplying the input by a negative 
connection value. Unfortunately, with a physical device, the connection may only 
30 take on zero or positive values under such a scenario 
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[00114] In other words, either there can be a connection or no connection. 
A connection can simulate a negative connection by dedicating a particular 
connection to be negative, but one connection cannot begin positive and through 
a learning process change to a negative connection. In general, if it starts 
5 positive, it can only go to zero. In essence, it is the idea of possessing a 
negative connection initially that results in the simulation, because this does not 
occur in a biological network. Only one type of signal travels through axons and 
dendrites in a biological network. That signal is transferred into the flow of a 
neurotransmitter whose effect on the receiving neuron can be either excitatory or 
10 inhibitory, depending on the neuron, thereby dedicating certain connections 
inhibitory and excitatory 

[00115] One method for solving this problem is to utilize two sets of 
connections for the same output, having one set represent the positive 
connections and the other set represent the negative connections. The output of 
15 these two layers can be compared, and the layer with the greater output will 
output either a high signal or a low signal, depending on the type of connection 
set (inhibitory or excitatory). Such a configuration can be seen in FIG. 5, where 
the excitatory output can be, for example, a layer 1 output and the inhibitory 
output can be a layer 2 output. 

20 [001 16] A truth table for the output of circuit 700 is illustrated at block 780 

in FIG. 7. As indicated at block 780, when an excitatory output is high and the 
inhibitory output is also high, the final output is low. When the excitatory output is 
high and the inhibitory output is low, the final output is high. Similarly, when the 
excitatory output is low and the inhibitory output is high, the final output is low. 

25 When the excitatory output is low and the inhibitory output is also low, the final 
output is low. Note that layers 704 and 708 may thus comprise excitatory 
connections, while layers 706 and 710 may comprise inhibitory connections. 

[00117] At all times during the learning process, a weak alternating electric 
field can be applied perpendicular to the connections. This can cause the 
30 connections to weaken by rotating the nanotube perpendicular to the connection 
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direction. This weakening of connections is important because it can allow for a 
much higher degree of adaptation. To understand this, one must realize that the 
connections cannot (practically) keep getting stronger and stronger. By 
weakening those connections not contributing much to the desired output, we 
5 decrease the necessary strength of the needed connections and allow for more 
flexibility in continuous training. Other mechanisms, such as increasing the 
temperature of the nanotube suspension could also be used for such a purpose. 

[00118] The circuit depicted in FIG. 7 can be separated into two separate 
circuits. The first part of the circuit can be composed of nanotube connections, 
10 while the second part of the circuit comprises the "neurons" and the learning 
mechanism (i.e., op-amps/comparator). The learning mechanism on first glance 
appears similar to a relatively standard circuit that could be implemented on 
silicon with current technology. Such a silicon implementation can thus comprise 
the "neuron" portion of the chip. 

15 [00119] The second part of the circuit (i.e., the connections) is thus a new 

type of chip structure, although it could be constructed with current technology. 
The connection chip can be composed of an orderly array of electrodes spaced 
anywhere from, for example, 100nm to 1/ym or perhaps even further. In a 
biological system, one talks of synapses connecting neurons. It is in the 

20 synapses where the information is processed, (i.e., the "connection weights"). 
Similarly, such a chip can contain all of the synapses for the physical neural 
network. A possible 2-diminsional arrangement thereof can be seen in FIG. 8. 

[00120] The training of such a chip is primarily based on two assumptions. 
First, the inherent parallelism of a physical neural network (i.e., a Knowm™ 
25 network or system) can permit all training sessions to occur simultaneously, no 
matter how large the associated connection network. Second, recent research 
has indicated that near perfect aligning of nanotubes can be accomplished in no 
more than 15 minutes utilizing practical voltages of about 5V. 
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[00121] If one considers that the input data, arranged as a vector of binary 
"high's" and "low's" is presented to the Knowm™ network or system 
simultaneously, and that all training vectors are presented one after the other in 
rapid succession (e.g., perhaps 100 MHz or more), then each connection would 
5 "see" a different frequency in direct proportion to the amount of time that its 
connection is required for accurate data processing (i.e., provided by a feedback 
mechanism). Thus, if it only takes approximately 15 minutes to attain an almost 
perfect state of alignment, then this amount of time would comprise the longest 
amount of time required to train, assuming that all of the training vectors are 
10 presented during that particular time period and adequate feedback has been 
provided. 

[00122] FIG. 9 illustrates a flow chart 900 of logical operational steps that 
can be followed to construct a connection network, in accordance with an 
embodiment of the present invention. Initially, as indicated at block 902, a 

15 connection gap is created from a connection network structures. As indicated 
earlier, the goal for such a connection network is generally to develop a network 
of connections of "just" the right values to satisfy particular information 
processing requirements, which is precisely what a neural network accomplishes. 
As illustrated at block 904, a solution is prepared, which is composed of 

20 nanoconductors and a "solvent." Note that the term "solvent" as utilized herein 
has a variable meaning, which includes the traditional meaning of a "solvent," 
and also a suspension. 

[00123] The solvent utilized can comprise a volatile liquid that can be 
confined or sealed and not exposed to air. For example, the solvent and the 

25 nanoconductors present within the resulting solution can be sandwiched between 
wafers of silicon or other materials. If the fluid has a melting point that is 
approximately at operating temperature, then the viscosity of the fluid could be 
controlled easily. Thus, if it is desired to lock the connection values into a 
particular state, the associated physical neural network (i.e., a Knowm™ network 

30 or system) can be cooled slightly until the fluid freezes. The term "solvent" as 
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utilized herein thus can include fluids such as for example, toluene, hexadecane, 
mineral oil, liquid crystals, etc. Note that the solution in which the nanoconductors 
(i.e., nanoconnections) are present should generally comprise a substance that 
does not conduct electricity and allows for the suspension of nanoparticles. 

5 [00124] Thus, when the resistance between the electrodes is measured, 

the conductivity of the nanoconductors can be measured, not that of the solvent. 
The nanoconductors can be suspended in the solution or can alternately lie on 
the bottom surface of the connection gap if the gap is 2-diminsional (i.e., formed 
on a planar surface such as electrodes deposited on the surface of a substrate). 
10 Note that the solvent described herein may also comprise liquid crystal media. 
Cabron nanotube alignment is possible, for example, by dissolving nanotubes in 
liquid crystal media, such that liquid crystals thereof align with an electric field 
and take the nanotubes and/or other nanoconductors with them. 

[00125] As illustrated thereafter at block 906, the nanoconductors must be 
15 suspended in the solvent, either dissolved or in a suspension of sorts, but 
generally free to move around, either in the solution or on the bottom surface of 
the gap. As depicted next at block 908, the electrical conductance of the solution 
must be less than the electrical conductance of the suspended nanoconductor(s). 
Next, as illustrated at block 910, the viscosity of the substance should not be too 
20 much so that the nanoconductors cannot move when an electric field (e.g., 
voltage) is applied across the electrodes. Finally, as depicted at block 912, the 
resulting solution of the "solvent" and the nanoconductors is thus located within 
the connection gap. 

[00126] Note that although a logical series of steps is illustrated in FIG. 9, 
25 it can be appreciated that the particular flow of steps can be re-arranged. Thus, 
for example, the creation of the connection gap, as illustrated at block 902, may 
occur after the preparation of the solution of the solvent and nanoconductor(s), 
as indicated at block 904. FIG. 9 thus represents merely possible series of steps, 
which can be followed to create a connection network. It is anticipated that a 
30 variety of other steps can be followed as long as the goal of achieving a 
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connection network in accordance with the present invention is achieved. Similar 
reasoning also applies to FIG. 10. 

[00127] FIG. 10 illustrates a flow chart 1000 of logical operations steps 
that can be followed to strengthen nanoconductors within a connection gap, in 
5 accordance with a preferred of the present invention. As indicated at block 1002, 
an electric field can be applied across the connection gap discussed above with 
respect to FIG. 9. The connection gap can be occupied by the solution discussed 
above. As indicated thereafter at block 1004, to create the connection network, 
the input terminals can be selectively raised to a positive voltage while the output 
10 terminals are selectively grounded. 

[00128] As illustrated thereafter at block 1006, connections thus form 
between the inputs and the outputs. The important requirements that make the 
resulting physical neural network functional as a neural network is that the longer 
this electric field is applied across the connection gap, or the greater the 
15 frequency or amplitude, the more nanoconductors align and the stronger the 
connection becomes. Thus, the connections that experience the most feedback 
during training become the strongest. 

[00129] As indicated at block 1008, the connections can either be initially 
formed and have random resistances or no connections will be formed at all. By 

20 forming initial random connections, it might be possible to teach the desired 
relationships faster, because the base connections do not have to be built up as 
much. Depending on the rate of connection decay, having initial random 
connections could prove to be a faster method, although not necessarily. A 
connection network will adapt itself to whatever is required regardless of the 

25 initial state of the connections. 

[00130] Thus, as indicated at block 1010, as the electric field is applied 
across the connection gap, the more the nonconductor(s) will align and the 
stronger the connection becomes. Connections (i.e., synapses) that are not used 
are dissolved back into the solution, as illustrated at block 1012. As illustrated at 
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block 1014, the resistance of the connection can be maintained or lowered by 
selective activations of the connections. In other words, "if you do not use the 
connection, it will fade away," much like the connections between neurons in a 
human brain in response to Long Term Depression, or LTD. 

5 [00131] The neurons in a human brain, although seemingly simple when 

viewed individually, interact in a complicated network that computes with both 
space and time. The most basic picture of a neuron, which is usually 
implemented in technology, is a summing device that adds up a signal. Actually, 
this statement can be made even more general by stating that a neuron adds up 

10 a signal in discrete units of time. In other words, every group of signals incident 
upon the neuron can be viewed as occurring in one moment in time. Summation 
thus occurs in a spatial manner. The only difference between one signal and 
another signal depends on where such signals originate. Unfortunately, this type 
of data processing excludes a large range of dynamic, varying situations that 

15 cannot necessarily be broken up into discrete units of time. 

[00132] The example of speech recognition is a case in point. Speech 
occurs in the time domain. A word is understood as the temporal pronunciation 
of various phonemes. A sentence is composed of the temporal separation of 
varying words. Thoughts are composed of the temporal separation of varying 

20 sentences. Thus, for an individual to understand a spoken language at all, a 
phoneme, word, sentence or thought must exert some type of influence on 
another phoneme, word, sentence or thought. The most natural way that one 
sentence can exert any influence on another sentence, in the light of neural 
networks, is by a form of temporal summation. That is, a neuron "remembers" 

25 the signals it received in the past. 

[00133] The human brain can accomplish such a feat in an almost trivial 
manner. When a signal reaches a neuron, the neuron has an influx of ions rush 
through its membrane. The influx of ions contributes to an overall increase in the 
electrical potential of the neuron. Activation is achieved when the potential inside 
30 the cell reaches a certain threshold. The one caveat is that it takes time for the 
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cell to pump out the ions, something that it does at a more or less constant rate. 
So, if another signal arrives before the neuron has time to pump out all of the 
ions, the second signal will add with the remnants of the first signal and achieve 
a raised potential greater than that which could have occurred with only the 
5 second signal. The first signal influences the second signal, which results in 
temporal summation. 

[00134] Implementing this in a technological manner has proved difficult in 
the past. Any simulation would have to include a "memory" for the neuron. In a 
digital representation, this requires data to be stored for every neuron, and this 

10 memory would have to be accessed continually. In a computer simulation, one 
must discritize the incoming data, since operations (such as summations and 
learning) occur serially. That is, a computer can only do one thing at a time. 
Transformations of a signal from the time domain into the spatial domain require 
that time be broken up into discrete lengths, something that is not necessarily 

15 possible with real-time analog signals in which no point exists within a time- 
varying signal that is uninfluenced by another point. 

[00135] A physical neural network, however, is generally not digital. A 
physical neural network is a massively parallel analog device. The fact that 
actual molecules (e.g., nanoconductors) must move around (in time) makes one 

20 form of temporal summation a natural occurrence. This temporal summation is 
built into the nanoconnections and can occur at a time scale much longer that 
that which is possible with capacitors and standard analog circuitry in micron 
dimension VLSI designs. The easiest way to understand this is to view the 
multiplicity of nanoconnections as one connection with one input into a neuron- 

25 like node (Op-amp, Comparator, etc.). This can be seen in FIG. 1 1 . 

[00136] FIG. 11 illustrates a schematic diagram of a circuit 1100 
demonstrating temporal summation within a neuron, in accordance with one 
embodiment of the present invention. As indicated in FIG. 11, an input 1102 can 
be provided to nanoconnections 1104, which in turn provides a signal, which can 
30 be input to an amplifier 1110 (e.g., op amp) at node B. A resistor 1106 can be 
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connected to node A, which in turn is electrically equivalent to node B. Node B 
can be connected to a negative input of amplifier 1100. Resistor 1108 can also 
be connected to a ground 1108. Amplifier 1110 can provide output 1114. Note 
that although nanoconnections 1104 is referred to in the plural it can be 
5 appreciated that nanoconnections 1 104 can comprise a single nanoconnection or 
a plurality of nanoconnections. For simplicity sake, however, the plural form is 
used to refer to nanoconnections 1104. 

[00137] Input 1102 can be provided by another physical neural network 
(i.e., a Knowm™ network or system) to cause increased connection strength of 
10 nanoconnections 1 104 over time. This input will most likely arrive in pulses, but 
can also be continuous, depending upon a desired implementation. A constant 
or pulsed electric field perpendicular to the connections would serve to constantly 
erode the connections, so that only signals of a desired length or amplitude could 
cause a connection to form. 

15 [00138] Once the connection is formed, the voltage divider formed by 

nanoconnection 1104 and resistor 1106 can cause a voltage at node A in direct 
proportion to the strength of nanoconnections 1 104. When the voltage at node A 
reaches a desired threshold, the amplifier (i.e., an op-amp and/or comparator), 
will output a high voltage (i.e., output 1114). The key to the temporal summation 

20 is that, just like a real neuron, it takes time for the electric field to breakdown the 
nanoconnections 1 104, so that signals arriving close in time will contribute to the 
firing of the neuron (i.e., op-amp, comparator, etc.). Temporal summation has 
thus been achieved. The parameters of the temporal summation could be 
adjusted by the amplitude and frequency of the input signals and the 

25 perpendicular electric field. 

[00139] FIG. 12 illustrates a block diagram illustrating a pattern recognition 
system 1200, which can be implemented with a physical neural network device 
1222, in accordance with an alternative embodiment of the present invention. 
Note that the pattern recognition system 1200 can be implemented as a speech 
30 recognition system. Although a pattern recognition system 1200 is depicted 
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herein in the context of speech recognition, a physical neural network device 
(i.e., a Knowm™ device) can be implemented in association with other types of 
pattern recognition systems, such as visual and/or imaging recognition systems. 

[00140] FIG. 12 thus is not considered a limiting feature of the present 
5 invention but is presented for general edification and illustrative purposes only. 
The diagram depicted in FIG. 12 can, of course, be modified as new applications 
and hardware are developed. The development or use of a pattern recognition 
system such as pattern recognition system 1200 of FIG. 12 by no means limits 
the scope of the physical neural network (i.e., Knowm™) disclosed herein. 

[00141] FIG. 12 illustrates in block diagram fashion, a system structure of 
a speech recognition device using a neural network according to one alternative 
embodiment of the present invention. The pattern recognition system 1200 
depicted in FIG. 12 can be provided with a CPU 1211 (e.g., a microprocessor) for 
performing the functions of inputting vector rows and instructor signals (vector 
rows) to an output layer for the learning process of a physical neural network 
device 1222, and changing connection weights between respective neuron 
devices based on the learning process. Pattern recognition system 1200 can be 
implemented within the context of a data-processing system, such as, for 
example, a personal computer or personal digital assistant (PDA), both of which 
are well known in the art. 

[00142] The CPU 1211 can perform various processing and controlling 
functions, such as pattern recognition, including but not limited to speech and/or 
visual recognition based on the output signals from the physical neural network 
device 1222. The CPU 1211 is connected to a read-only memory (ROM) 1213, a 
25 random-access memory (RAM) 1214, a communication control unit 1215, a 
printer 1216, a display unit 1217, a keyboard 1218, an FFT (fast Fourier 
transform) unit 1221, a physical neural network device 1222 and a graphic 
reading unit 1224 through a bus line 1220 such as a data bus line. The bus line 
1220 may comprise, for example, an ISA, EISA, or PCI bus. 
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[00143] The ROM 1213 is a read-only memory storing various programs 
or data used by the CPU 1211 for performing processing or controlling the 
learning process, and speech recognition of the physical neural network device 
1222. The ROM 1213 may store programs for carrying out the learning process 
5 according to error back-propagation for the physical neural network device or 
code rows concerning, for example, 80 kinds of phonemes for performing speech 
recognition. The code rows concerning the phonemes can be utilized as second 
instructor signals and for recognizing phonemes from output signals of the 
neuron device network. Also, the ROM 1213 can store programs of a 
10 transformation system for recognizing speech from recognized phonemes and 
transforming the recognized speech into a writing (i.e., written form) represented 
by characters. 

[00144] A predetermined program stored in the ROM 1213 can be 
downloaded and stored in the RAM 1214. RAM 1214 generally functions as a 

15 random access memory used as a working memory of the CPU 1211. In the 
RAM 1214, a vector row storing area can be provided for temporarily storing a 
power obtained at each point in time for each frequency of the speech signal 
analyzed by the FFT unit 1221. A value of the power for each frequency serves 
as a vector row input to a first input portion of the physical neural network device 

20 1222. Further, in the case where characters or graphics are recognized in the 
physical neural network device, the image data read by the graphic reading unit 
1224 are stored in the RAM 1214. 

[00145] The communication control unit 1215 transmits and/or receives 
various data such as recognized speech data to and/or from another 

25 communication control unit through a communication network 1202 such as a 
telephone line network, an ISDN line, a LAN, or a personal computer 
communication network. Network 1202 may also comprise, for example, a 
telecommunications network, such as a wireless communications network. 
Communication hardware methods and systems thereof are well known in the 

30 art. 
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[00146] The printer 1216 can be provided with a laser printer, a bubble- 
type printer, a dot matrix printer, or the like, and prints contents of input data or 
the recognized speech. The display unit 1217 includes an image display portion 
such as a CRT display or a liquid crystal display, and a display control portion. 
5 The display unit 1217 can display the contents of the input data or the recognized 
speech as well as a direction of an operation required for speech recognition 
utilizing a graphical user interface (GUI). 

[00147] The keyboard 1218 generally functions as an input unit for varying 
operating parameters or inputting setting conditions of the FFT unit 1221, or for 
10 inputting sentences. The keyboard 1218 is generally provided with a ten-key 
numeric pad for inputting numerical figures, character keys for inputting 
characters, and function keys for performing various functions. A mouse 1219 
can be connected to the keyboard 1218 and serves as a pointing device. 

[00148] A speech input unit 1223, such as a microphone can be 
15 connected to the FFT unit 1221. The FFT unit 1221 transforms analog speech 
data input from the voice input unit 1223 into digital data and carries out spectral 
analysis of the digital data by discrete Fourier transformation. By performing a 
spectral analysis using the FFT unit 1221, the vector row based on the powers of 
the respective frequencies are output at predetermined intervals of time. The FFT 
20 unit 1221 performs an analysis of time-series vector rows, which represent 
characteristics of the inputted speech. The vector rows output by the FFT 1221 
are stored in the vector row storing area in the RAM 1214. 

[00149] The graphic reading unit 224, provided with devices such as a 
CCD (Charged Coupled Device), can be used for reading images such as 

25 characters or graphics recorded on paper or the like. The image data read by the 
image-reading unit 1224 are stored in the RAM 1214. Note that an example of a 
pattern recognition apparatus, which can be modified for use with the physical 
neural network of the present invention, is disclosed in non-limiting and non- 
essential U.S. Patent No. 6,026,358 to Tomabechi, February 16, 2000, "Neural 

30 Network, A Method of Learning of a Neural Network and Phoneme Recognition 
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Apparatus Utilizing a Neural Network." Non-essential and non-limiting U.S. 
Patent No. 6,026,358 is incorporated herein by reference, but only for general 
edification and illustrative purposes. 



[00150] The implications of a physical neural network are tremendous. 
5 With existing lithography technology, many electrodes in an array such as 
depicted in FIG. 5 or 14 can be etched onto a wafer of silicon. The "neurons" 
(i.e., amplifiers, diodes, etc.), as well as the training circuitry illustrated in FIG. 6, 
could be built onto the same silicon wafer. By building the neuron circuitry on 
one side of a substrate, and the electrode arrays on the other side, chips can be 

10 built that are no longer limited in synapse density. A solution of suspended 
nanoconductors could be placed between the electrode connections and the chip 
could be packaged. One could also place a rather large network parallel with a 
computer processor as part of a larger system. Such a network, or group of 
networks, can add significant computational capabilities to standard computers 

1 5 and associated interfaces. 

[00151] For example, such a chip can be constructed utilizing a standard 
computer processor in parallel with a large physical neural network or group of 
physical neural networks. A program can then be written such that the standard 
computer teaches the neural network to read, or create an association between 
20 words, which is precisely the same sort of task in which neural networks can be 
implemented. This would amount to nothing more than presenting the Knowm 
network with a pre-defined sequence of input and output patterns stored in 
memory. 

[00152] Once the physical neural network is able to "read", it can be 
25 taught for example to "surf 1 the Internet and find material of any particular nature. 
A search engine can then be developed that does not search the Internet by 
"keywords", but instead by meaning. This idea of an intelligent search engine has 
already been proposed for standard neural networks, but until now has been 
impractical because the network required was too big for a standard computer to 
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simulate. The use of a physical neural network as disclosed herein now makes a 
truly intelligent search engine possible. 

[001 53] A physical neural network can be utilized in other applications, 
such as, for example, speech recognition and synthesis, visual and image 
5 identification, management of distributed systems, self-driving cars and filtering. 
Such applications have to some extent already been accomplished with standard 
neural networks, but are generally limited in expense, practicality and not very 
adaptable once implemented. 

[00154] The use of a physical neural network can permit such applications 
10 to become more powerful and adaptable. Indeed, anything that requires a bit 
more "intelligence" could incorporate a physical neural network. One of the 
primary advantages of a physical neural network is that such a device and 
applications thereof can be very inexpensive to manufacture, even with present 
technology. The lithographic techniques required for fabricating the electrodes 
15 and channels there between has already been perfected and implemented in 
industry. 

[00155] Most problems in which a neural network solution is generally 
implemented are complex adaptive problems, which change in time. An example 
is weather prediction. The usefulness of a physical neural network is that it could 

20 handle the enormous network needed for such computations and adapt itself in 
real-time. An example wherein a physical neural network (i.e., Knowm™) can be 
particularly useful is the Personal Digital Assistant (PDA). PDA's are well known 
in the art. A physical neural network applied to a PDA device can be 
advantageous because the physical neural network can ideally function with a 

25 large network that could constantly adapt itself to the individual user without 
devouring too much computational time from the PDA processor. A physical 
neural network could also be implemented in many industrial applications, such 
as developing a real-time systems control to the manufacture of various 
components. This systems control can be adaptable and totally tailored to the 

30 particular application, as necessarily it must. 
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[00156] The training of multiple connection networks between neuron 
layers within a multi-layer neural network is an important feature of any neural 
network. The addition of neuron layers to a neural network can increase the 
ability of the network to create increasingly complex associations between inputs 
5 and outputs. Unfortunately, the addition of extra neuron layers in a network 
raises an important question: How does one optimize the connections within the 
hidden layers to produce the desired output? 

[001 57] The neural network field was stalled for some time trying to 
answer this question until several parties simultaneously stumbled onto a 

10 computationally efficient solution, now referred to generally as "back- 
propagation" or "back-prop" for short. As the name implies, the solution involves 
a propagation of error back from the output to the input. Essentially, back- 
propagation amounts to efficiently determining the minimum of an error surface 
composed of n variables, where the variable n represents the number of 

15 connections. 

[00158] Because back propagation is a computational algorithm, this 
concept may not make much sense physically. Another related question to ask 
is can the neurons in a human brain take a derivative? Do they "know" the result 
of a connection on another neuron? In other words, how does a neuron know 
20 what the desired output is if each neuron is an independent summing machine, 
only concerned with its own activation level and firing only when that activation is 
above threshold? What exactly can a neuron "know" about its environment? 

[00159] Although this question is certainly open for debate, it is plausible 
to state that a neuron can only "know" if it has fired and whether or not its own 

25 connections have caused the firing of other neurons. This is precisely the 
Hebbian hypothesis for learning: "if neuron A repeatedly takes part in firing 
neuron B, then the connection between neuron A and B strengthens so that 
neuron A can more efficiently take part in firing neuron B". With this hypothesis, 
a technique can be derived to train a multi-layer physical neural network device 

30 without utilizing back-propagation or any other training "algorithm", although the 
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technique mirrors back-propagation in form, as information is transferred from 
output layers to input layers, providing feedback in the form of pulses that modify 
connections. 

[00160] In fact, the resulting Knowm™ (i.e., physical neural network) can 
5 be self-adaptable and does not require any calculations. In other words, the 
network and its training mechanism can be a physical process that arises from 
feedback signals within the network. The structure of a Knowm™ physical neural 
network or synapse thereof thus creates a situation in which learning simply 
takes place when a desired output is given. The description that follows is thus 
10 based on the use of a physical neural network (i.e., a Knowm™) and constituent 
nanoconnections thereof. 

[00161] FIG. 13 illustrates a schematic diagram of a 2-input, 1 -output, 2- 
layer inhibitory physical neural network 1300, which can be implemented in 
accordance with one embodiment of the present invention. As indicated in FIG. 

15 13, two layers 1326 and 1356 of physical neural network 1300 can be 
distinguished from one another. Note that as utilized herein, the term "layer" can 
be defined as comprising a connection network. Such a connection network can 
include one or more neurons in association with a plurality of nanoconductors 
present in a solvent, as explained herein. In FIG. 13, layers 1326 and 1356 are 

20 respectively labeled L1 and L2. Inputs 1304 and 1306 to a connection network 
1302 are also indicated in FIG. 13, wherein inputs 1304 and 1306 are 
respectively labeled 11 and 12 and connection network 1302 is labeled C1. 

[00162] Inputs 1304 and 1302 (i.e. 11 and 12) generally provide one or 
more signals, which can be propagated through connection network 1302 (i.e., 

25 C1). Connection network 1302 thus generates a first output signal at node 1303 
and a second output signal at node 1305. The first output signal provided at 
node 1303 is further coupled to an input 1323 of an amplifier 1312, while the 
signal output signal provided at node 1305 is connected to an input 1325 of an 
amplifier 1314. Amplifier 1312 thus includes two inputs 1323 and 1311, while 

30 amplifier 1314 includes two inputs 1315 and 1325. Note that a voltage V t can be 
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measured at input 1311 to amplifier 1312. Similarly, voltage V, can also be 
measured at input 1315 to amplifier 1314. Additionally, a resistor 1316 can be 
coupled to node 1305 and a resistor 1310 is connected to node 1303. Resistor 
1310 is further coupled to a ground 1309. Resistor 1316 is further connected to 
5 ground 1309. Resistors 1310 and 1316 are labeled R b in FIG. 13. 

[00163] Amplifier 1312 can thus function as a neuron A and amplifier 1314 
functions as a neuron B. The two neurons, A and B, respectively sum the signals 
provided at nodes 1303 and 1305 to provide output signals thereof at nodes 
1319 and 1321 (i.e., respectively H1 and H2). Additionally, a switch 1308, which 

10 is labeled S1, is connected between nodes 1303 and 1319. Likewise, a switch 
1322, which is also labeled S1, is connected between nodes 1305 and 1321. A 
resistor 1318 is coupled between an output of amplifier 1312 and node 1319. 
Similarly, a resistor 1320 is coupled between an output of amplifier 1314 and 
node 1321. Node 1319, which carries signal H1, is connected to a connection 

15 network 1328. Also, node 1321, which carries signal H2, is connected to 
connection network 1328. 

[00164] Note that connection network 1328 is labeled C2 in FIG. 3. A first 
signal can be output from connection network 1328 at node 1331. Likewise, a 
second signal can be output from connection network 1328 at node 1333. A 

20 resistor 1330, which is labeled R b , is coupled between node 1331 and ground 
1309. Also, a resistor 1334, which is also labeled R b , is connected between node 
1333 and ground 1309. Node 1333 is further connected to an input 1353 to 
amplifier 1338, while node 1331 is further coupled to an input 1351 to amplifier 
1336. Note that resistor 1330 is also coupled to input 1351 at node 1331, while 

25 resistor 1 334 is connected to input 1 353 at node 1 333. 

[00165] A voltage V t can be measured at an input 1335 to amplifier 1336 
and an input 1337 to amplifier 1338. Amplifiers 1335 and 1338 can be 
respectively referred to as neurons C and D. An output from amplifier 1336 is 
connected to a NOT gate 1340, which provides a signal that is input to a NOR 
30 gate 1342. Additionally, amplifier 1338 provides a signal, which can be input to 
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NOR gate 1342. Such a signal, which is output from amplifier 1338 can form an 
inhibitory signal, which is input to NOR gate 1342. Similarly, the output from 
amplifier 1336 can comprise an excitatory signal, which is generally input to NOT 
gate 1340. The excitatory and inhibitory signals respectively output from 
5 amplifiers 1336 and 1338 form an excitatory/inhibitory signal pair. NOR gate 
1342 generates an output, which is input to an amplifier 1344 at input node 1347. 
A voltage V d can be measured at input node 1346, which is coupled to amplifier 
1344. 

[00166] Thus, the signals H1 and H2, which are respectively carried at 
10 nodes 1319 and 1321, are generally propagated through connection network 
1328, which is labeled C2, where the signals are again summed by the two 
neurons, C and D (i.e., amplifiers 1336 and 1338). The output of these two 
neurons therefore form an excitatory/inhibitory signal pair, which through the 
NOT gate 1340 and the NOR gate 1342 are transformed into a signal output 01 
15 as indicated at output 1348. Note that signal output node 01 can be measured 
at input node 1347 of amplifier 1344. Amplifier 1344 also includes an output 
node 1349, which is coupled to node 1331 through a switch 1350, which is 
labeled S2. Output 1349 is further coupled to a NOT gate 1354, which in turn 
provides an output which is coupled to node 133 through a switch 1352, which is 
20 also labeled S2. 

[00167] For inhibitory effects to occur, it may be necessary to implement 
twice as many outputs from the final connection network as actual outputs. 
Thus, every actual output represents a competition between a dedicated 
excitatory signal and inhibitory signal. The resistors labeled R b (i.e., resistors 

25 1330 and 1334) are generally very large, about 10 or 20 times as large as a 
nanoconnection. On the other hand, the resistors labeled Rf (i.e., resistors 1318 
and 1 320) may possess resistance values that are generally less than that of a 
nanoconnection, although such resistances can be altered to affect the overall 
behavior of the associated physical neural network. V t represents the threshold 

30 voltage of the neuron while V d represents the desired output. S1 and S2 are 
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switches involved in the training of layers 1 and 2 respectively (i.e., L1 and L2, 
which are indicated respectively by brackets 1326 and 1356 in FIG. 13). 



[00168] For reasons that will become clear later, a typical training cycle 
can be described as follows. Initially, an input vector can be presented at 11 and 
5 12. For this particular example, such an input vector generally corresponds to 
only 4 possible combinations, 11, 10, 01 or 00. Actual applications would 
obviously require many more inputs, perhaps several thousand or more. One 
should be aware that the input vector does not have to occur in discrete time 
intervals, but can occur in real time. The inputs also need not necessarily be 
10 digital, but for the sake of simplicity in explaining this example, digital 
representations are helpful. While an input pattern is being presented, a 
corresponding output can be presented at V d . Again, in this particular case there 
is only one output with only two corresponding possible outcomes, 1 or 0. The 
desired output also does not have to be presented in discrete units of time. 

15 [00169] For learning to occur, the switches 1350 and 1352 (i.e., S2) can 

be closed, followed by switches 1308 and 1322 (i.e., S1). Both groupings of 
switches (S1 and S2) can then be opened and the cycle thereof repeated. 
Although only two layers L1 and L2 are illustrated in FIG. 13, it can be 
appreciated that a particular embodiment of the present invention can be 

20 configured to include many more layers. Thus, if more than two layers exist, then 
the switches associated with the preceding layer can be initially closed, then the 
second to last, the third to last and so on, until the last switch is closed on the 
input layer. The cycle is repeated. This "training wave" of closing switches can 
occur at a frequency determined by the user. Although it will be explained in 

25 detail later, the more rapid the frequency of such a training wave, the faster the 
learning capabilities of the physical neural network. 

[00170] For example, it can be assumed that no connections have formed 
within connection networks C1 or C2 and that inputs are being matched by 
desired outputs while the training wave is present. Since no connections are 
30 present, the voltage at neurons A, B, C and D are all zero and consequently all 
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neurons output zero. One can quickly realize that whether the training wave is 
present or not, a voltage drop will not ensue across any connections other than 
those associated with the input connection network. The inputs, however, are 
being activated. Thus, each input is seeing a different frequency. Connections 
5 then form in connection network C1 , with the value of the connections essentially 
being random. 

[00171] Before a connection has been made, the voltage incident on 
neurons A and B is zero, but after a connection has formed, the voltage jumps to 
approximately two diode drops short of the input voltage. This is because the 

10 connections form a voltage divider with R b , such that R b (i.e., resistors 1310 
and/or 1316) possesses a resistance very much larger than that of the 
nanoconnections. Two reasons for utilizing a large R b are to minimize power 
consumption of the physical neural network during a normal operation thereof, 
and to lower the voltage drop across the connections so that learning (i.e. 

15 connection modification) only takes place when feedback is present. Fortunately, 
nanotube contact resistances are on the order of about 100 kQ, or more, which 
can allow for an R b of a few MQ or greater. V t must be somewhere between two 
diode drops of the input voltage and the voltage produce by one nanoconnection 
in a voltage divider with R b , the later being lower than the former. 

20 [00172] Once connections have formed across C1 and grown sufficiently 

strong enough to activate neurons A and B, the connections across C2 can form 
in the same manner. Before continuing, however, it is important to determine 
what will occur to the nanoconnections of connection network 1302 (i.e., C1) 
after they grow strong enough to activate the first layer neurons. For the sake of 

25 example, assume that neuron A has been activated. When S1 is closed in the 
training wave, neuron A "sees" a feedback that is positive (i.e., activated). This 
locks the neuron into a state of activation, while S1 is closed. Because of the 
presence of diodes in connection network 1302 (i.e., C1), current can only flow 
from left to right in C1. This results in the lack of a voltage drop across the 

30 nanoconnections. 
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[00173] If another electric field is applied at this time to weaken the 
nanoconnections (e.g., perhaps a perpendicular field), the nanoconnections 
causing activation to the neuron can be weakened (i.e., the connections running 
from positive inputs to the neuron are weakened) This can also be accomplished 
5 by an increased temperature, which could naturally arise from heat dissipation of 
the other circuitry on the chip. This feedback will continue as long as the 
connections are strong enough to activate the neuron (i.e., and no connections 
have formed in the second layer). Nanoconnections can thus form and be 
maintained at or near the values of neuron activation. This process will also 
10 occur for ensuing layers until an actual network output is achieved. 

[00174] Although the following explanation for the training of the newly 
formed (and random) connections may appear unusual with respect to FIG. 13, 
the configuration depicted in FIG. 13 represents the smallest, simplest network 
available to demonstrate multi-layer training. A typical physical neural network 
15 can actually employ many more inputs, outputs and neurons. In the process of 
explaining training, reference is made to FIG. 13, however, embodiments of the 
present invention can be implemented with more than simply two inputs and one 
output. 

[00175] FIG. 13 is thus presented for illustrative purposes only and the 
20 number of inputs, outputs, neurons, layers, and so forth, should not be 
considered a limiting feature of the present invention, which is contemplated to 
cover physical neural networks that are implemented with hundreds, thousands, 
and even millions of such inputs, outputs, neurons, layers, and so forth. Thus, 
the general principles explained here with respect to FIG. 13 can be applied to 
25 physical neural networks of any size. 

[00176] It can be appreciated from FIG. 13 that neuron C (i.e., amplifier 
1336) is generally excitatory and neuron D (i.e., amplifier 1338) is generally 
inhibitory. The use of NOT gates 1340 and 1354 and NOR gate 1342 create a 
situation in which the output is only positive if neuron C is high and neuron D 
30 zero (i.e., only if the excitatory neuron C is high and the inhibitory neuron D low). 
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For the particular example described herein with respect to FIG. 13, where only 
one output is utilized, there generally exists a fifty-fifty chance that the output will 
be correct. 

[00177] Recall, however, that in a typical physical neural network many 
5 more outputs are likely to be utilized. If the output is high when the desired 
output is low, then the training neuron (i.e., amplifier 1344, the last neuron on the 
right in FIG. 13) outputs a high signal. When S2 is closed during the training 
wave, this means that the post connections of the excitatory neuron will receive a 
high signal and the post connections of the inhibitory neuron a negative signal 
10 (i.e., because of the presence of NOT gate 1354). Note that through feedback 
thereof, each neuron will be locked into each state while S2 is closed. 

[00178] Because of the presence of diodes within connection network 
1328 (i.e., C2), there will be no voltage drop across those connections going to 
the excitatory neuron. There will be a voltage drop, however, across the 

15 nanoconnections extending from positive inputs of C2 to the inhibitory neuron 
(i.e., amplifier 1338). This can result in increases in inhibitory nanoconnections 
and a decrease in excitatory nanoconnections thereof (i.e., if an eroding is 
present). This is exactly what is desired if the desired output is low when the 
actual output is high. A correspondingly opposite mechanism strengthens 

20 excitatory connections and weakens inhibitory connections if the desired output 
is high when the actual output is low. When the desired output matches the 
actual output, the training neurons output is dependent on the gain of the 
differential amplifier. 

[00179] Thus far an explanation has been presented describing how the 
25 last layer of a physical neural network can in essence train itself to match the 
desired output. An important concept to realize, however, is that the activations 
coming from the previous layer are basically random. Thus, the last connection 
network tries to match essentially random activations with desired outputs. For 
reasons previously explained, the activations emanating from the previous layer 
30 do not remain the same, but fluctuate. There must then be some way to "tell" the 
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layers preceding the output layer which particular outputs are required so that 
their activations are no longer random. 

[00180] One must realize that neurons simply cannot fire unless a neuron 
in a preceding layer has fired. The activation of output neurons can be seen as 
5 being aided by the activations of neurons in previous layers. An output neuron 
"doesn't care" what neuron in the previous layer is activating it, so long as it is 
able to produce the desired output. If an output neuron must produce a high 
output, then there must be at least one neuron in the previous layer that both has 
a connection to it and is also activated, with the nanoconnection(s) being strong 
10 enough to allow for activation, either by itself or in combination with other 
activated neurons. 

[00181] With this in mind, one can appreciate that the nanoconnections 
associated with pre-output layers can be modified. Again, by referring to FIG. 
13, it can be appreciated that when S2 is closed (and S1 still open), Rf may form 

15 a voltage divider with the connections of C2, with R b taken out of the picture. 
Recall that R f represents resistors 1318 and/or 1320, while R b represents 
resistors 1310 and/or 1316. Because of the diodes on every input and output of 
C2, only connections that go from a positive activation of neurons A and B to 
ground after C2 will allow current to flow. Recall as explained previously that 

20 only those nanoconnections that are required to be strengthened in the output 
connection matrix thereof will be negative, so that the voltage signals H1 and H2 
measured respectively at nodes 1319 and 1321 are the direct result of how many 
neurons "need" to be activated in the output layer. In other words, the more 
neurons in layer i+1 that need activation, the lower the total equivalent resistance 

25 of all connections connecting a neuron in layer T and the neurons in layer "i+1" 
needing activation. 

[00182] By thereafter closing S1, the previous layer neurons in essence 
"know" how much of their activation signal is being utilized. If their signal is being 
utilized by many neurons in a preceding layer, or by only a few with very strong 
30 nanoconnections, then the voltage that the neuron receives as feedback when 
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S1 is closed decreases to a point below the threshold of the neuron. Exactly 
what point this occurs at is dependent on the value of R f (i.e., resistors 1318 
and/or 1320) As Rf becomes larger, less resistance is generally required to lower 
H1 or H2 to a point below the threshold of the neuron. This feedback voltage is 
5 very important, as this is how the network matches inputs with desired outputs. 
First, note that the feedback is local, confined to individual neurons. In essence, 
if a neuron needs to supply activations to many neurons, then it must strengthen 
its connections to neurons that are activating it, so that it may continue to do its 
job. 

10 [00183] Several subtleties exist in this feedback process. Although the 

feedback voltage is largely determined by the neurons' pre-synaptic connections, 
(i.e., "axonal" connections), it is also determined by the neurons' post synaptic 
connections (i.e., dendritic connections). If the feedback voltage, Vf, is lower 
than the threshold voltage, Vt, then the dendritic connections will be 

15 strengthened. Because the feedback voltage is a function of both the axonal and 
dendritic connections, one scenario that cannot lower Vf below Vt is weak axonal 
connections and very strong dendritic connections. In other words, if the 
dendritic connections (to activated neurons) are very strong, then the axonal 
connections (to neurons needing activation) must be correspondingly stronger. 

20 This relationship is not linear. Thus, based on the foregoing, nanoconnections in 
layers preceding the output layer can modify themselves. 

[00184] Referring again to FIG. 13 as an example, if the voltage at H1 
decreases to a point below V t when S1 is closed, then either neuron C or D (or 
both) will require the activation of neuron A to achieve the desired output. When 
25 S1 closes, neuron A receives the voltage at H1 as feedback, which is below the 
threshold of the neuron. This causes the neuron to output zero, which can again 
be transmitted by feedback to the neuron's input. Now the neuron is locked in a 
feedback loop constantly outputting zero. This causes an electric field to be 
generated across the connections of C1, from positive activations of 11 and/or 12 
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(i.e., inputs 1304 and/or 1306) to neuron A. Now the nanoconnections causing 
the activation of neuron A are even stronger. 



[00185] Note that connections could also form between activated pre- 
synaptic neurons and the neuron in question even if no initial connection is 
5 present, or if the post-synaptic neuron is inactivated. This last form of connection 
formation is important because it allows for a form of connection exploration. In 
other words, connections can be formed, and if the feedback mechanism finds it 
useful to match a desired input-output relationship, it will be strengthen. If not, it 
will be weakened. This allows neuron A to keep outputting a high signal that in 
10 turn allows the output neurons to match the desired output. The same argument 
can apply for neuron B, or any neuron in any layer preceding the output layer. 

[00186] Although a detailed description of the process has been provided 
above, it is helpful to view the process from a generalized perspective. Again, 
assuming that no connections are present in any of the connection networks, 

15 assume that a series of input vectors are presented to the inputs of the network, 
and a series of output vectors are presented to the desired output, while the 
training wave is present. The training wave should be at a frequency equal or 
greater than the frequency at which input patterns are presented or otherwise the 
first few layers will not be trained and the network will be unable to learn the 

20 associations. The first layer connection network, analogous to C1 in FIG. 13, will 
begin to form connections, and continue to build connections until the sum of the 
connection hovers around the activation threshold for the succeeding neurons 
(amplifiers). Once C1 connections have been created, C2 connections can be 
created in the same manner, this time with the input signals coming from the 

25 neuron activations of the preceding neurons. 

[00187] The connections can, just like C1, build up and hover around the 
threshold voltage for the succeeding neurons. This pattern of forming 
connections can generally occur until a signal is achieved at the output. Once a 
signal has been outputted, the feedback process begins and the training wave 
30 guides the feedback so that connections are modified strategically, from the 
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output connection network to the input connection network, to achieve the 
desired output. The training is continued until the user is satisfied with the 
networks ability to correctly generate the correct output for a given input. 

[00188] In evaluating a standard feed-forward multi-layer neural network, 
5 connections generally form between every neuron in one layer and every neuron 
in the next layer. Thus, neurons in adjacent layers are generally completely 
interconnected. When implementing this in a physical structure where 
connection strengths are stored as a physical connection, the architecture must 
be configured that allows for both total connectedness between layers and which 
10 also provides for the efficient use of space. In a physical neural network device 
(i.e., a "Knowm™ device), connections form between two conducting electrodes. 
The space between the electrodes can be filled with a nano-conductor/dielectric 
solvent mixture, which has been described previously herein. As an electric field 
is applied across the electrode gap, connections form between the electrodes. A 
1 5 basic method and structure for generating a large number of synapses on a small 
area substrate is illustrated in FIG. 14. 

[00189] FIG. 14 illustrates a pictorial diagram of a perspective view of a 
system 1400 that includes a synapse array 1401, which can be implemented in 
accordance with one embodiment of the present invention. The synapse array 
1401 illustrated in FIG. 14 can be implemented as a chip, which may also be 
referred to as a Knowm™ chip or a physical neural network chip. Additionally, 
the configuration depicted in FIG. 14 can be referred to simply as a "synapse" 
chip. The use of the term "synapse" as utilized herein is thus analogous to use of 
the term synapse in the biological arts. Although not biological in nature, the 
functions of a synapse or synapse chip as described herein do have similarities 
to biological systems. A synapse is simply the point at which a nerve impulse is 
transmitted from one neuron to another. Similarly, a synapse chip can be 
configured as the point at which electrical signals are transmitted from artificial 
neuron to another. 



20 



25 
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[00190] The basic structure of a physical neural network device, such as a 
physical neural network chip and/or synapse chip, is depicted in FIG. 14. 
Synapse array 1401 (i.e., a synapse chip) can formed from a substrate 1404. By 
forming a gap 1402 between two plates P1 and P2 covered with electrodes, filled 
5 with a solution of nano-conductors and a dielectric solvent, it can be appreciated 
that connections can easily form between every input and every output by 
aligning vertically from one input electrode to a perpendicular output electrode. It 
is thus apparent that the input and output electrodes would include some sort of 
conducting material. 

10 [00191] The input electrodes are indicated in FIG. 14 by input electrodes 

11, 12, 13, 14 and 15. The output electrodes are indicated in FIG. 14 by output 
electrodes 01, 02, 03, 04, and 05. For a Knowm™ device (e.g., a synapse 
chip), a perpendicular field can be applied across the connection gap to weaken 
the connections, so that the connection strengths are fully controllable. Various 

15 placements of auxiliary electrodes, either on P1, P2, or both can accomplish this 
feature. Alternatively, the temperature could be maintained at an elevated level 
so that thermal energy can break down connections. This last form, (i.e., 
temperature degradation), could provide the most elegant solution. During the 
learning phase, an increased voltage drop across the connections can result in 

20 substantial heat generation within the chip. This heat, in turn, can be vital to the 
learning process by weakening connections that \are not used. 

[00192] FIG. 15 illustrates a pictorial diagram 1600 of a perspective view 
of an alternative chip structure 1601 with parallel conductors on output, which 
can be implemented in accordance with an alternative embodiment of the 

25 present invention. As indicated in FIG. 15, the actual chip layout can be seen as 
two basic chip structures, an input layer 1606 and an output layer 1604, each 
sandwiched over a gap 1602 filled with a nanoconductor/dielectric solvent 
mixture. The output layer 1604 can generally be formed from output electrodes 
01, 02, 03, and 04, while the input layer can be formed from input electrodes 

30 11,12, 13, and 14. 
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[00193] Although only four input electrodes and four output electrodes are 
illustrated in FIG. 15, this particular number of input and output electrodes is 
depicted for illustrative purposes only. In a typical synapse chip implemented in 
accordance with the present invention, many more (i.e., thousands, millions, etc.) 
5 input and output electrodes can be utilized to form input and output electrode 
arrays thereof. Additionally, the nanoconductors form connections in the 
intersections between input and output electrodes due to the increased electric 
field strength. Chip structure 1601 thus represents one type of a synapse chip, 
which can be implemented in accordance with one possible embodiment of the 
10 present invention described herein. 

[00194] FIG. 16 illustrates a perspective view of a system 1700 that 
includes a connection formation 1701, in accordance with a preferred or 
alternative embodiment of the present invention. As depicted in FIG. 16, 
nanoconnections 1702 can form at intersections between input and output 
15 electrodes due to an increase in electric field strength. Architectures of this type 
can offer substantial benefits for producing a Knowm™ synapse chip. These 
include ease of assembly and efficient use of space. Regarding the ease of 
assembly, the total chip can comprise two electrode arrays aligned perpendicular 
to each other, with a layer of nano-conductor/dielectric solution between the two. 

20 [00195] FIG. 17 illustrates a system 1750 illustrating the use of system 

1700 of FIG. 16 in the context of a synapse chip and neural network 
configuration thereof. System 1750 indicates a chip 1758 along with a top chip 
layer 1752 and a both chip layer 1754, which are respectively indicated through 
the use of solid lines (representing layer 1752) and dashed lines (representing 

25 layer 1754). A diagram 1756 represents connection conduits, while a schematic 
diagram 1756 represents graphically the mathematical operations taking place 
via chip 1758. Note that in FIGS. 16 and 17, like or analogous parts or elements 
are indicated by identical reference numerals. 

[00196] A larger view of an adaptive network system can thus be seen in 
30 FIG. 17. As previously mentioned, a network can be constructed by integrating 
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many base neurons (i.e., see schematic diagram 1756). Each base neuron can 
contain both temporal and a spatial summation of signals generated by other 
base neurons. This summed signal can then be compared to a threshold 
voltage, and if the summed voltage exceeds the threshold voltage, a pulse may 
5 be emitted at the base neurons pre-synaptic electrodes. The inverse of the pre- 
synaptic pulse can also be emitted at the base neurons post-synaptic electrodes. 

[00197] The base neurons can be in a perpendicular array structure (i.e., 
chip 1758) composed of two or more layers 1752, 1754 coupled with Knowm 
synapses (i.e., system 1700). Each Knowm synapse can be composed of 

10 connection conduits, separated by a characteristic distance "d", where each 
connection conduit is the result of nano-particles aligning in an electric field 
generated by the temporal and sequential firing of the coupled base neurons 
(i.e., see schematic diagram 1756). External inputs to the network can be 
coupled to any post-synaptic electrode of any base neuron in any layer. And any 

1 5 network output can be provided at any pre-synaptic electrode of any base neuron 
in any layer. 

[00198] Other attempts at creating a neural-like processor require 
components to be placed precisely, with resolutions of a nanometer. The design 
of FIG. 16, for example, only requires two perpendicular electrode arrays. 

20 Prepared nanoconductors, such as, for example, nanotubes and/or nanowires, 
can be simply mixed with a dielectric solvent. A micro-drop of the solution can 
thereafter be placed between the electrode arrays. Regarding the efficient use of 
space, even with electrode widths of 1 micron and spacing between electrodes of 
2 microns, 11 million synapses or more could potentially fit on 1 square 

25 centimeter. If electrode widths of 100nm, with spacing of 200nm, are utilized, 
approximately 1 billion synapses could potentially fit on 1 cm 2 . 

[00199] Although the electrode dimensions cannot be lowered indefinitely 
without a considerable loss in connection resistance variation, it is conceivable 
that a 1cm 2 chip could hold over 4 billion synapses (e.g., 50nm electrodes and 
30 100nm spacing = 4.4 billion synapes/cm 2 ). Because neuron circuitry could 
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potentially be constructed on the other side of the synapse arrays, very compact 
neural processors with high neuron/synapse density could also be constructed. 

[00200] Some considerations about the construction of a chip should be 
addressed. For example, the distance between the input electrodes should 
5 generally remain at a distance close, but not touching, the output electrodes. If 
carbon nanotubes are utilized for the nano-conductors within the gaps, one 
would need to prepare the nanotubes to lengths shorter than the gap distance. If 
the gap distance is, for example, approximately, 100nm, then the nanotubes 
should be sized less this dimension. Given a diameter of about 1.5nm, 
10 nanotubes can only go so far, perhaps 10's of nanometers. At 1.5 nm, one is 
now approaching atomic distances. The distance between the two electrodes 
could be maintained by resting the upper plate of electrodes on "pedestals", 
which could be formed by an interference photolithography technique already 
developed. 

15 [00201] Note that as utilized herein, the term "chip" generally refers to a 

type of integrated circuit, which is known in the art as a device comprising a 
number of connected circuit elements such as transistors and resistors, 
fabricated on a single chip of silicon crystal or other semiconductor material. 
Such chips have traditionally been manufactured as flat rectangular or square 

20 shaped objects. It can be appreciated, however, that such chips can be 
fabricated in a variety of shapes, including circular and spherical shapes in 
addition to traditional square, box or rectangular shaped integrated circuit chips. 
Thus, a synapse chip or physical neural network chip (i.e., a Knowm™ chip) can 
also be fabricated as a spherical integrated circuit. 

25 [00202] An example of a spherical chip is disclosed in non-limiting and 

non-essential U.S. Patent No. 6,245,630, "Spherical Shaped Semiconductor 
Circuit," which issued to Akira Ishikawa of Ball Semiconductor, Inc. on June 12, 
2001, and is incorporated herein by reference. The spherical chip disclosed in 
U.S. Patent No. 6,245,630 generally comprises a spherical shaped 

30 semiconductor integrated circuit ("ball") and a system and method for 
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manufacturing the same. Thus, a ball can replace the function of a flat, 
conventional chip. 



[00203] The physical dimensions of the ball allow it to adapt to many 
different manufacturing processes which otherwise could not be used. 
5 Furthermore, the assembly and mounting of the ball may facilitate efficient use of 
the semiconductor as well as circuit board space. Thus, a physical neural 
network chip and/or synapse chip as disclosed herein can be configured as such 
a ball-type chip in addition to a rectangular or square shaped integrated circuit 
chip. 

10 [00204] Based on the foregoing it can be appreciated that embodiments 

are generally directed toward a physical neural network synapse chip and also a 
method for forming such a synapse chip. The synapse chip disclosed herein with 
respect to one or more embodiments can be configured to include an input layer 
comprising a plurality of input electrodes and an output layer comprising a 

15 plurality of output electrodes, such that the output electrodes are located above 
or below the input electrodes. A gap is generally formed between the input layer 
and the output layer. A solution can then be provided which is prepared from a 
plurality of nanoconductors and a dielectric solvent. 

[00205] The solution can be located within the gap, such that an electric 
20 field is applied across the gap from the input layer to the output layer to form 
nanoconnections of a physical neural network implemented by the synapse chip. 
Such a gap can thus be configured as an electrode gap. The input electrodes 
can be configured as an array of input electrodes, while the output electrodes 
can be configured as an array of output electrodes. 

25 [00206] The nanoconductors can form nanoconnections at one or more 

intersections between the input electrodes and the output electrodes in 
accordance with an increase in strength of the electric field applied across the 
gap from the input layer to the output layer. Additionally, an insulating layer can 
be associated with the input layer, and another insulating layer associated with 
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the output layer. The input layer can be formed from a plurality of parallel N-type 
semiconductors and the output layer formed from a plurality of parallel P-type 
semiconductors. 

[00207] Similarly, the input layer can be formed from a plurality of parallel 
5 P-type semiconductors and the output layer formed from a plurality of parallel N- 
type semiconductors. Thus, the nanoconnections can be strengthened or 
weakened respectively according to an increase or a decrease in strength of the 
electric field. As an electric field is applied across the electrode gap, 
nanoconnections thus form between the electrodes. 

10 [00208] The most important aspect of the electrode arrays described 

herein is their geometry. Generally, any pattern of electrodes in which almost 
every input electrode is connected to every output electrode, separated by a 
small gap, is a valid base for a connection network. What makes this particular 
arrangement better than other arrangements is that it is very space-efficient. By 

15 allowing the connection to form vertically, a third dimension can be utilized, 
consequently gaining enormous benefits in synapse density. 

[00209] To understand just how space-efficient a Knowm™ chip utilizing 
connection formation in a third-dimension could be, consider the NET talk 
network created by Terry Sejnowski and Charles Rosenberg in the mid 1980's. 
20 NET talk took the text-representation of a word and could output the phonemic 
representation, thereby providing a text-to-speech translation. Such a network 
provided 203 inputs, 120 hidden neurons and 26 outputs, for a total of 
approximately 28 thousand synapses. 

[00210] Utilizing electrode widths of approximately 200nm and spacing 
25 between electrodes of approximately 400nm, one could contain 28 thousand 
synapses on about 10160(jm 2 . In comparison, a conventional synapse, including 
all of the weight storage resistors and switches, I0-I4 current mirrors, multiplier 
and sign switching circuitry takes up approximately 106x1 13|jm or 1 1978 pm 2 . In 
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other words, one could fit 28 thousand synapses in less than the area previously 
required to store only one synapse. 

[00211] The benefits of creating a neural network processor are thus 
great. The ability to implement as many as 1 billion synapses on 1cm 2 of surface 
5 substrate is a tremendous leap forward over prior art neural network 
technologies. Another innovation is the ability to mass-produce pre-trained, 
large-scale neural network chips. A physical neural network as disclosed herein 
does not have to be taught at all, but can instead be manufactured with the 
desired connections already in place. This is an important feature for consumer 
10 devices. For example, in most cellular telephones produced today, the ability to 
recognize rudimentary speech is available. One might, after pre-recording a 
voice, speak the word "Dave" and the cellular telephone can automatically call 
Dave after matching the word just spoken to a list of other pre-recorded names 
and thereafter pick the best match. 

15 [00212] This is a rather rudimentary form of pattern recognition and could 

therefore be replaced by an exceedingly small Knowm™ synapse chip. For 
example, a Knowm™ chip can be taught at the factory to translate speech into 
text, thereby eliminating the need to pre-record ones voice for recognition tasks 
and instead relying on a more general speech recognition technique. Once the 

20 factory Knowm™ chip is trained, the synapse resistance values can be 
determined. With knowledge of what each synapse value needs to be, one can 
then design a perpendicular array chip so that the electrode widths create a 
cross-sectional area inversely proportional to the resistance of each synapse. 

[00213] In other words, the resistance of each connection is generally a 
25 function of the cross-sectional area of the connection between electrodes. By 
pre-forming the electrodes to certain specified widths, and then allowing the 
maximum number of connections to form at each electrode intersection, a 
physical neural network can be mass-produced. Such a configuration can allow 
a very general network function (e.g., voice or facial recognition) to be produced 
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and sold to consumers, without the necessity of forcing the consumer to train the 
network. FIG. 18 below illustrates this concept. 



[00214] FIG. 18 illustrates a schematic diagram of a system 1800 of 
electrode widths encoding specific synapses resistances, in accordance with an 
5 alternative embodiment of the present invention. As indicated in FIG. 18, a 
plurality of bottom layer electrodes 1810, 1812, and 1814 having different cross 
sections are located below a plurality of top layer electrodes 1802, 1804, 1806 
and 1808. After the physical neural network or synapse chip is assembled, the 
maximum number of connections can be formed at each synapse, which is 
10 equivalent to the desired resistance at each synapse. Of course, the function 
relating to the cross-section area of the electrodes and the corresponding 
resistance will differ from substance to substance and will most likely have to be 
determined experimentally. 

[00215] A synapse or physical neural network chip could therefore be 
15 produced with certain ready-made abilities, such as voice or facial recognition. 
After installation, it is up to the designer to create a product that can then modify 
itself further and continue to adapt to the consumer. This could undoubtedly be 
an advantageous ability. Utilizing the example of the cellular telephone, the 
cellular telephone could in essence adapt its speech-recognition to the accent or 
20 manner of speech of the individual user. And all of this is possible because the 
Knowm™ synapses are so space-efficient. Networks with very powerful pattern 
recognition abilities could fit into a tiny fraction of a hand-held device, such as, for 
example, a wireless personal digital assistant and/or a cellular telephone. 

[00216] The embodiments indicated herein are thus directed toward a 
25 physical neural network that can be configured from a connection or a plurality of 
connections of molecules or molecular conductors (e.g., molecules), such as 
nanoconductors, such as, for example, nanowires, nanotubes, and/or 
nanoparticles. Such physical neural network can be implemented in the form of 
one or more synapse chips that can be combined with a neuron system (e.g., a 
30 neuron chip) of independent summing circuits. 
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[00217] The fundamental concept of a Knowm™ network or system (e.g., 
a Knowm™ synapse chip) is remarkably simple. When particles in a dielectric 
solution are exposed to an electric field (i.e., AC or DC), the particles align with 
the field. As the particles align, the resistance between the respective electrodes 
5 decreases. The connection becomes stable once the electric field is removed. 
As the strength or frequency of the applied electric field is increased, the 
connections become increasingly aligned and the resistance further decreases. 
By applying a perpendicular electric field, one can also decrease the strength of 
the connections. Such connections can be utilized as "synapses" in a physical 
10 neural network chip (also referred to as a synapse chip), and the result is a 
Knowm™ chip - a fully adaptable, high-density neural network chip. 

[00218] Although a multi-layer, feed-forward structure, has been 
addressed, recurrent, unsupervised neural networks can provide many 
advantages. An adaptive neuron, as a base for a larger Hebbian-based 
15 recurrent network utilizing Knowm™ synapses will be discussed. The basic 
theory behind adaptive networks will be presented; after which will be described 
the appropriate translation to an electrical system that utilizes Knowm™ 
synapses. 

[00219] A Knowm™ synapse can be configured in a manner that is highly 
20 appropriate for an adaptive neural network, which can also be referred to as an 
adaptive integration network or simply, an adaptive network. As indicated earlier, 
adaptive neural networks to date have been limited to software designs and/or 
conventional hardware implementations. Adaptive neural networks have not 
been designed or implemented based on nanotechnology components, systems, 
25 and/or networks as discussed herein. 

[00220] FIG. 19 illustrates a schematic diagram of one example of an 
adaptive integration network 1900, comprising six interconnected processing 
elements, neurons 1910, 1920, 1930, 1940, 1950, and 1960. Although the 
adaptive integration network 1900 is illustrated as containing six neurons on a 
30 two-dimensional plane, it is to be understood that the present invention is not 
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limited to the particular number of the neurons or to any particular network 
topology. In fact, implementations of an adaptive integration network may 
comprise hundreds, thousands, even millions of interconnected neurons. 
Neurons may be arranged in various physical and logical configurations, 
5 including but not limited to hexagonal, circular, rectangular, toroidal structures in 
one, two, three, or higher dimensions. 

[00221] Such neurons can be implemented as a Knowm™ network or 
system. Each neuron 1910, 1920, 1930, 1940, 1950, and 1960 can be 
individually formed from standard photolithography or alternate procedures by 

10 building circuits capable of neuronal function, as will be later discussed. 
Alternatively, connections between neurons 1910, 1920, 1930, 1940, 1950, and 
1960 may be formed as nanoconductor(s) suspended within a dielectric solvent 
or solution. An example of nanoconnections that may be implemented to 
neurons 1910, 1920, 1930, 1940, 1950, and 1960 is provided by 

1 5 nanoconnections 304 of FIG. 3. 

[00222] A neuron combined with pre-synaptic electrodes can thus be the 
basic processing element of an adaptive integration network and can be 
(although not necessarily) configured to receive signals from its "pre-synaptic" 
neurons as input and, in response, transmit output signals to its "post-synaptic" 
20 neurons. A neuron has two output states, firing or non-firing. In one embodiment, 
binary output signal values of Xj =1 and Xj =0 are assigned for the firing and non- 
firing states, some embodiments may employ non-binary values for the output 
signal Xj, for example, within a range 0.0 < Xj < 1 .0. 

[00223] As another example, the value of the output signal Xj is 0.0 if the 
25 neuron is not firing, and greater than or equal to 1.0 if the neuron is firing as 
explained in more detail herein after. In the context of electronic circuitry, the 
output of the neuron would comprise a voltage. When a neuron fires, that neuron 
could potentially cause its post-synaptic neurons to fire, as more specifically 
explained herein after, which could cause their post-synaptic neurons to fire, and 
30 so on, setting up a chain reaction along an active pathway. 
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[00224] Any neuron in an adaptive integration network can be designated 
as a data input neuron or a data output neuron. A data input neuron is a neuron 
that receives a signal external to the adaptive integration network, and a data 
output neuron is a neuron whose output signal is transmitted to a destination 
5 external to the adaptive integration network. Accordingly, external signals input 
into data input neurons may initiate a chain reaction of neuron firings throughout 
the adaptive integration network. When the neuron firings eventually affect the 
state of the data output neurons, the output of the adaptive integration network 
will change in response. 

10 [00225] In the example of FIG. 19, neurons 1910 and 1920 are data input 

neurons because neurons 1910 and 1920 receive external input signals 1902 
and 1904, respectively. Neuron 1950 is a data output neuron because neuron 
1950, when firing, produces an output signal 106. In this configuration, an 
asserted input signal 1902 eventually causes neuron 1910 to fire, which may 

15 then cause neuron 1940 and then neuron 1950 to fire, thereby producing the 
output signal 1906. Thus, the adaptive integration network 100 produces an 
output signal 1906 in response to an input signal 1902. In many implementations, 
it is convenient for data input neurons to only receive a single external signal and 
no internal signals as input. 

20 [00226] In an adaptive neural network, a connection is the conduit along 

which a neuron receives a signal from another neuron. Connections can be 
formed between neurons in any direction to transmit a signal from an output of a 
pre-synaptic neuron to an input of a post-synaptic neuron. Typically, a neuron 
plays both roles, first as a post-synaptic neuron for receiving input signals from 

25 its pre-synaptic neurons, and second as a pre-synaptic neuron for generating 
output signals to its post-synaptic neurons. 

[00227] For example, with continued reference to FIG. 19, pre-synaptic 
neuron 1910 is coupled to post-synaptic neuron 1940 by connection 1914, thus 
neuron 1910 is configured to transmit information to neuron 1940. In FIG. 19, 
30 neuron 1910 is also coupled to neuron 1920 by connection 1912; neuron 1920 is 
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coupled to neuron 1930 by connection 1923; neuron 1930 is coupled to neuron 
1940 by connection 1934 and to neuron 1960 by connection 1936; neuron 1940 
is coupled to neuron 1950 by connection 1945; neuron 1950 is coupled to neuron 
1910 by connection 1951 and to neuron 1960 by connection 1956; and neuron 
5 1960 is coupled to neuron 1910 by connection 1961 and to neuron 1920 by 
connection 162. 

[00228] Connections (e.g., nanoconnections) may be excitatory or 
inhibitory, through which transmitted signals respectively promote or retard the 
firing of the postsynaptic neuron in response. With continued reference to FIG. 
10 19, excitatory a fully connected arrow represents connections, and inhibitory 
connections are illustrated with an offset, blocked arrow. For example, 
connections 1914, 1923, 1936, 1945, 1951, and 1962 are excitatory, and 
connections 1912, 1934, 1956, and 1961 are inhibitory. 

[00229] Excitatory connections are used to transmit signals from one 
15 neuron to another in a feedback loop or other active pathway. Inhibitory 
connections, on the other hand, prevent neurons from firing and are useful in 
providing internal regulation among feedback loops, but cannot actually form a 
connection in a feedback loop. In the context of a physical neural network, the 
inhibitory connections may cause a momentary increase in the threshold voltage 
20 of the post-synaptic neuron, thereby inhibiting the activations of the neuron. 

[00230] An adaptive integration network may be configured to include 
feedback loops. A loop is a closed circuit of linked excitatory connections 
arranged in the same circular direction. For example, adaptive integration 
network 1900 comprises two loops, a first loop with neurons 1910, 1940, and 
25 1950 indicated with black excitatory connections 1914, 1945, and 1951, and a 
second loop with neurons 1920, 1930, and 1960 denoted with gray excitatory 
connections 1923, 1936, and 1962. 

[00231] Loops are highly interactive with other loops. In general, a loop 
can be mutually reinforcing or mutually competitive with one or more other loops. 
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The adaptive integration network 1900 depicted in FIG. 1 illustrates an example 
with two mutually competitive loops. If an input signal 1910 is applied causing 
neuron 1910 to fire in the first (black) loop, then a chain reaction is set up 
wherein neuron 1940 fires, then neuron 1950 fires, then neuron 1910 fires again, 
5 and so forth. 

[00232] In addition, neurons 1910 and 1950 have inhibitory connections 
1912 and 1956, respectively for suppressing firings of neurons 1920 and 1960, 
respectively, in the second (gray) loop. Thus, activation of the first (black) loop 
can force the deactivation of the second (gray) loop. Similarly, activating the 
10 second (gray) loop builds a circular chain of firings through neurons 1920, 1930, 
and 1960, while suppressing activity in neurons 1910 and 1940, via inhibitory 
connections 1961 and 1934, respectively. 

[00233] Mutually interacting loops may be aggregated to form metaloops 
at a higher level of integration. For example, two mutually interacting loops may 
15 share one or more connections in common such that activity in one loop will 
affect the activity in the other loop. Referring to FIG. 20, a portion of an adaptive 
integration network 2000 is depicted with two mutually interacting loops 2002 and 
2004. 

[00234] Loop 2002 comprises six neurons 2010, 2020, 2030, 2040, 2050, 
20 and 2060 connected in sequence, and loop 2004 comprises five neurons 250, 
2060, 2070, 2080, and 2090 connected in sequence. Both loop 2002 and 2004 
share neurons 2050 and 2060, which are coupled by connection 2056. Activity 
on either loop influences activity on the other loop. For example, if neuron 2010 
in loop 2002 fires, that firing eventually results in the firing of neuron 2060, which 
25 transmits a signal to neuron 2010 of loop 2002 and to neuron 2070 of loop 2004. 
Similarly, if neuron 2070 in loop 2004 fires, that firing eventually results in the 
firing of neuron 2060, which transmits a signal to neuron 2010 of loop 2002 and 
to neuron 2070 of loop 2004. 
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[00235] As another example, one loop could branch off an active pathway 
to another loop, thereby initiating activity in the other loop. FIG. 21 illustrates a 
portion of an adaptive integration network 2100 with two loops 2102 and 2104. 
Loop 2102 comprises three neurons 2110, 2120, and 2130 connected in 
5 sequence, and loop 2104 comprises three neurons 2140, 2150, and 2160 
connected in sequence. Furthermore, loop 2102 is connected to loop 2104 by a 
connection 2134 from neuron 2130 of loop 2102 to neuron 2140 of loop 2104. 
Activity in loop 2102, eventually results in the firing of neuron 2130, which 
sustains the activity of loop 2102 by transmitting an output signal to neuron 21 10 
10 of loop 2102 and initiates activity in loop 304 by transmitting the output signal via 
connection 2134 to neuron 340 of loop 2104. 

[00236] Since an adaptive integration network provides much flexibility in 
configuration, it is to be understood that the present invention is not limited to any 
particular configuration of neurons and connections. Preferably, it is desirable to 
15 choose the number, distribution, and types of connections to maximize the total 
number of feedback loops while minimizing the functional constraints and 
interdependence of the loops. In general, this goal can be met by employing as 
many connections per node as feasible for a given implementation. 

[00237] The distribution of connections can vary from implementation to 
20 implementation of an adaptive integration network. For example, a maximum 
length can limit connections so that distant neurons are not directly connected, 
and the assignment of connections can be determined randomly or in 
accordance with an algorithm designed to give each neuron a similar physical or 
logical arrangement. 

25 [00238] In an adaptive integration network, a neuron fires in response to 

firings of the neuron's pre-synaptic neurons under certain conditions. More 
specifically, each neuron has an associated excitation level G, which is 
responsive to the signals received from the neuron's pre-synaptic neurons. The 
neuron can fire when the neuron's excitation level G is greater than or equal to 
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the neuron's threshold value, 0. In the context of a physical neural network, this 
is accomplished with an integrator circuit, as will be described. 

[00239] Furthermore, each connection can be characterized by a 
corresponding synaptic efficiency in transferring its signal, represented by a 
5 connection weight Wj, where i indicates the i th connection for the neuron. In the 
context of a physical neural network, the synaptic efficiency is a direct result of 
the alignment of the nano-conductors between pre-synaptic and post-synaptic 
electrodes, and the alignment is in turn a result of heightened activation. 

[00240] In a hardware implementation, when a pre-synaptic neuron fires a 
10 signal to its post-synaptic neurons, the firing neuron causes the excitation level G 
of the post-synaptic neurons to change by a factor directly related to the 
properties of an integrator, as discussed previously. After firing, the neuron's 
excitation level G is reset to a base level. In a hardware implementation, a 
refractory pulse generator, as discussed, can accomplish this previously. If the 
15 neuron does not fire, on the other hand, the integrator preserves the neuron's 
excitation level G, so that the excitation level G may accumulate over time and 
the neuron may eventually fire. In one embodiment, however, the excitation level 
G is subject to a decay process, for example, by multiplying the current excitation 
level by an attenuation parameter in the range 0.0 £ a <1.0. In a hardware 
20 implementation, this could be accomplished, for example, by storing charge from 
synaptic activations in a capacitor and allowing for a small leakage current that 
serves the function of the attenuation parameter. 

[00241] In one embodiment, neurons may be subject to a refractory period 
in which the neuron's excitation level G is forced to remain at the base level for a 

25 given period of time. During the refractory period, the activity of its pre-synaptic 
neurons does not affect the neuron's excitation level G. Consequently, the 
refractory period can serve to impose a limit on the maximum firing rate of the 
neuron. As previously discussed, the refractory pulse generator triggers the 
grounding of all post-synaptic electrodes, thereby playing a crucial role in 

30 network learning. 
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[00242] In a hardware implementation, the following sub-circuits that 
compose an individual neuron accomplish the refractory period. The inputs from 
synaptic activations are summed via an integrator, which allows the accumulation 
of signals over time. The integrated signal is passed to a threshold circuit, such 
5 as a comparator or operational amplifier that outputs a high or low voltage in 
response to the integrator signal being above a set threshold. This signal is 
passed to a circuit that allows a pulse of period "T" to be generated. The output 
pulse is the output of the neuron. This output is feed into a refractory pulse 
generator, which serves the purpose of grounding the post-synaptic electrodes in 
10 a Knowm synapse while the neuron is actively generating a pulse. 

[00243] If the output pulse of the neuron was high, then the refractory 
pulse generator could comprise a NOT gate, for example. The grounding of the 
postsynaptic electrodes serves two purposed. First, the neuron is re-set to a 
zero level activation, as described earlier. Second, the lowered potential causes 
15 an increase in the electric field across all connection in a connection network 
currently activating the neuron. In other words, during the time of the refractory 
pulse, all the connections that are coming from firing neurons become stronger. 

[00244] Training is generally the process of updating the nano- 
connections in an adaptive integration network so that the adaptive integration 
20 network produces desired outputs in response to inputs. In contrast with prior 
techniques involving artificial neural networks that employ distinct training and 
implementation phases, training the adaptive integration network is constantly 
occurring during the normal operation of the adaptive integration network and is a 
direct result of feedback within the network. 

25 [00245] Prior to operation of the adaptive integration network, the 

connection weights within the adaptive integration network are initialized, for 
example, either randomly or to a preset value. During the operation of the 
adaptive integration network, the connection weights are constantly strengthened 
or weakened, provided that the connection weight strengthening or weakening 

30 conditions are met. Connection weight strengthening refers to the process of 
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decreasing the resistance of the nano-connection. Connection weight 
strengthening occurs whenever any two connected neurons fire in close temporal 
proximity, with the post-synaptic neuron firing after the pre-synaptic neuron, 
during the post-synaptic neurons refractory pulse period. 

5 [00246] Optionally in some adaptive network implementations, connection 

weight strengthening occurs every time a neuron fires, but the magnitude of the 
connection weight strengthening is a function of the amount of time since the pre- 
synaptic neuron of the connection has fired. This is a natural result of frequency 
dependence on connection formation in a Knowm synapse since connections 
10 contributing less to the over-all activation of a neuron will receive fewer 
"refractory" pulses and consequently see a decreased frequency of electric field 
across the pre and post-synaptic electrode terminals. 

[00247] Connection weight strengthening allows for frequently used 
neuronal pathways to be reinforced. As one neuron fires, the neuron produces an 

15 output signal that may induce one or more of the neuron's post-synaptic neurons 
to fire in close temporal proximity, thereby strengthening the connection between 
the neurons. Similarly, the firing of the post-synaptic neuron may cause that 
neuron's post-synaptic neuron to fire, creating a chain reaction of firing neurons 
along an active pathway. Since a connection weight increases when both the 

20 pre-synaptic and the post-synaptic neurons fire in close temporal proximity, each 
time the active neural pathway is used, the connection weights along the active 
pathway are increased. 

[00248] A loop is a special case of a frequently used active pathway, 
because, once initiated, the neurons in the loop successively fire in cycles 
25 around the loop. Each time the neurons fire, their connections are strengthened, 
yielding a stable loop circuit. Consequently, the connection weight strengthening 
rules foster stable circuits of self-reinforcing loops, which can constitute stored 
memory of patterns and other information 
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[00249] Connection weight weakening generally refers to the process of 
decreasing the strength of the connection. In an adaptive network, connection 
weight weakening occurs after a specified period of passivity for the connection. 
A connection is considered "passive" for particular point in time if the post- 
5 synaptic neuron and the pre-synaptic neuron of the connection have not fired in 
close temporal proximity in that period of time. Thus, the connection weights for 
passive connections progressively weaken, reducing the influence of those 
passive connections in the adaptive integration network. In a physical Knowm 
implementation, a decrease in synapse activations results in a lower frequency of 
10 applied electric field and thus a decrease in connection formation. As discuss 
previously, The connection formation could be constantly degraded by a 
perpendicular electric field or even from a dissolution process within the solution. 

[00250] Larger connection weights are slowly decreased, thereby allowing 
for strong connections to remain more or less fixed, slow to decay even after 

15 prolonged passivity. This effect is naturally achieved in a knowm network by a 
decrease in the local electric filed around a strong nano-connection, thereby 
weakening effects from perpendicular electric fields. Alternatively, a strong nano- 
connection results in higher van-der-wall attractions and a corresponding 
heightened resistance to dissolution within the dielectric medium. In an adaptive 

20 integration network, connection weights are constantly being adjusted during 
normal operation, for example, strengthened when two connected neurons fire in 
close temporal proximity or weakened after a period of passivity. Therefore, even 
mere use of the adaptive integration network causes the adaptive integration 
network to be fine-tuned. 

25 [00251] In certain cases, however, it is desirable to cause the adaptive 

integration network to learn and adapt to new patterns and information. FIG. 23 
illustrates a flowchart 2300 illustrating the operation of adaptive learning in 
accordance with one embodiment of the present invention. As illustrated in FIG. 
23, adaptive learning can be fostered by presenting input data to the adaptive 
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integration network, as indicated at block 2301. The input data causes neuron 
firings, leading to output data from output data neurons as the result. 

[00252] As indicated at decision block 2302, a loop can be controlled long 
as the output data does not match the desired output. The network activity of the 
5 adaptive integration network can be increased, as depicted at block 2304, which 
causes the output data to change. When the desired data is produced, the 
network activity is restored to a normal level, as described at block 2306. Various 
techniques may be employed to increase network activity, i. e. the rate of neural 
firings, including threshold lowering and neural output signal magnification. 

10 [00253] Network activity can be increased by lowering the thresholds of 

the neurons in the adaptive integration network. For example, the thresholds 
could be reduced by a fixed amount or proportion, such as to one half. Threshold 
lowering causes neurons to fire sooner, because the excitation level G only 
needs to reach a lower threshold level. Consequently, firing rate of neurons in the 

15 adaptive integration network is increased and, hence, the network activities of the 
adaptive integration network. 

[00254] Yet another technique for increasing network activity is to increase 
the magnitude of the neural signals. Each time a neuron fires, the excitation level 
G of the post-synaptic neurons are increased by a much larger amount because 
20 the neural output signal Xj is larger. Consequently, the threshold level of the 
neuron is reached much more rapidly, increasing the firing rate of neurons in the 
adaptive integration network and, hence, the network activity. This can be 
accomplished by increasing the supply voltage of the neuron circuitry while 
keeping the threshold voltage constant. 

25 [00255] Increasing network activity enables for new active pathways to be 

explored. For example, a neuron that is adjacent to an active pathway, but not 
part of the active pathway, might not ordinarily fire because it has low connection 
strength for a connection to a neuron on the active pathway. In this case, the 
excitation level G of the neuron does not sufficiently accumulate to the ordinary 
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threshold level to fire, for example, due to a more rapid attenuation of the 
excitation level G or to competing inhibitory inputs. A lowered threshold, however, 
may be low enough or the excitation level G may accumulate rapidly enough to 
induce that neuron to fire, enabling a new active pathway to be branched off the 
5 main active pathway. 

[00256] Increasing network activity can also cause an active pathway for 
one stable circuit to transform into an active pathway for another stable circuit. A 
stable circuit, which constitutes stored memory, information, or patterns within 
the adaptive integration network, represents a locally optimal position in the 
10 solution space (all possible outputs for all possible input). As a result, increasing 
network activity permits adaptive exploration through the solution space in search 
of other locally optimal positions for the new input/output data sets. Another 
result of increasing network activity is that the response time of the adaptive 
integration network is reduced, making the adaptive integration network faster. 

15 [00257] FIGS. 24 and 25 illustrate how increasing network activity can 

dismantle an active pathway. Note that in FIGS. 24 and 25, like or analogous 
parts are indicated by identical reference numerals. Thus, In FIG. 24, a system 
2400 includes an active pathway comprising neurons 2410, 2420, and 2430 with 
high connection weights of 0.7. The pathway that includes neurons 2440, 2450, 

20 and 2460 with low connection weights of 0.3, on the other hand, is inactive. 
Furthermore, the low connection weight of 0.3 for the connection between neuron 
2410 of the active pathway and neuron 2440 means that neuron 2440 rarely 
fires, because the connection weight is too low to cause the excitation level G2440 
of neuron 2440 to sufficiently increase to reach the ordinary threshold level. 

25 [00258] When network activity is increased, for example by lowering the 

threshold, the accumulated excitation level G 2 44o is now high enough with respect 
to the lowered threshold to cause neuron 2440 to fire in response to a firing of 
neuron 2410. When neuron 2440 fires, an output signal is transmitted to the 
neuron 2450, which also fires with the increased network activity. The firing of 
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neuron 2450 induces neuron 2460 to fire and therefore strengthen their 
connection. 

[00259] Neuron 2450, moreover, is the source of an inhibitory connection 
to neuron 2420 of the active pathway, which prevents neuron 2420 from firing so 
5 often. As both neuron 2450 and neuron 2420 fire, the inhibitory connection 
between the two neurons is strengthened, further preventing neuron 2420 from 
firing so often. Eventually, the passivity of neuron 2420 causes the connection 
between neuron 2410 and 2420 to weaken, completing the dislodging of the 
active pathway. 

10 [00260] FIG. 25 illustrates the result of dislodging the active pathway, in 

which the new active pathway comprises neurons 2410, 2440, 2450, and 2460. 
FIG. 25 thus illustrates a system 2500 in which neurons 2420 and 2430 were 
formerly part of an active pathway, but are no longer, because their connection 
weights have been weakened. Adaptive learning can be initiated in response to 

15 an external signal from a user when the output is wrong, which is analogous to a 
biological pain signal. This external signal causes the network activity to be 
increased, for example, by lowering the threshold levels of the neurons. The 
increased network activity causes the input signals to be deflected or rerouted 
onto new active pathways and loops, thereby exploring new stable circuits. 

20 [00261] These new pathways and loops will eventually affect the data 

output neurons and alter the output values. If the output values are still 
undesired, then the increase in the network activity is maintained, causing the 
new pathways and loops to be ephemeral and generating even newer active 
pathways and loops. As soon as the desired output is attained, the user can 

25 discontinue the network activity increasing signal, causing the relevant network 
parameters (thresholds, etc.) to rebound to their ordinary levels and ceasing the 
adaptive training. This process can be automated if the desired output is 
presented before hand so that the output of the adaptive integration network can 
be compared by computer with the desired output to generate the external signal. 
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[00262] In contrast with retraining methods for conventional artificial neural 
networks, including both software and hardware implementations thereof, 
adaptive learning with adaptive integration networks is less disruptive, particularly 
when implemented via nanotechnology devices and techniques, such as 
5 discussed herein. For example, with conventional artificial neural networks every 
neuron is perturbed during training, but with adaptive integration networks only 
the neurons along active pathways and their neighbors are affected. 

[00263] Thus, only relevant connections are adjusted, and previously 
established but unrelated loops and meta-loops are left intact, which hold 
10 previously learned classifications and information. Therefore, in further contrast 
with conventional artificial neural networks, a nanotechnology-based adaptive 
integration network can learn new information and patterns at any time without 
having to relearn previously learned material or going through a new training 
stage. 

[00264] FIG. 26 illustrates a flow chart 2600 of operations depicting logical 
operational steps for modifying a synapse of a physical neural network, in 
accordance with an alternative embodiment of the present invention. According 
to the operations generally illustrated in flow chart 2600 of FIG. 26, a Knowm™ 
synapse can be modified based on a neuron refractory period. The process is 
generally initiated, as indicated at block 2602. As depicted at block 2604, one or 
more signals can be output from a connection network formed for example, from 
nanoconnections, such as nanoconnections 304 of FIG. 3. 

[00265] Such signals may be generated in the form of a voltage or a 
current, depending upon a desired implementation. For illustrative purposes 
25 only, it can be assumed that such signals comprise voltage signals. As indicated 
next at block 2606, these signals provided by the connection network can be 
summed by at least one neuron within the physical neural network and then, as 
illustrated at block 2608, compared to a threshold value. The threshold voltage 
can be an externally applied and modifiable voltage. 
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[00266] If, as indicated at block 2610, the current state of activation does 
not exceed the threshold value or threshold voltage, the process simply 
terminates, as indicated at block 261 1 . If, however, the current state of activation 
does exceed the threshold value or threshold voltage, then the process 
5 continues, as indicated at block 2612, and a pulse (e.g., a voltage pulse or 
current pulse) is emitted from a neuron within the physical neural network. 
During this pulse, a "refractory pulse generator" grounds the postsynaptic 
junction thereof. 

[00267] This operation in turn can cause the synapses receiving pre- 
10 synaptic activation to experience an increase in the local electric field. The pre- 
synaptic electrodes of succeeding neurons and post-synaptic connections of the 
pulse emitting neuron thus receive a pulse, as indicated at block 2615. Thus, 
synapses that contribute to the activation of the neuron can receive an increase 
in the local electric field parallel to the connection direction and can also 
15 experience a higher frequency of activation, two parameters that increase the 
strength nanoconnections thereof, as indicated at block 2618, and thus the 
strength of the Knowm™ synapse. 

[00268] FIG. 27 illustrates a flow chart 2700 of operations illustrating 
logical operational steps for strengthening one or more nanoconnections of a 

20 connection network of a physical neural network by an increase in frequency, in 
accordance with an alternative embodiment of the present invention. Flow chart 
2700 of FIG. 27 generally illustrates a process for strengthening 
nanoconnections (e.g., nanconnections 304 of FIG. 3) of a physical neural 
network based on the close temporal proximity between two or more connected 

25 firing neurons. The process can be initiated, as indicated at block 2702, in which 
an initial (e.g., i th ) neuron, is fired. 

[00269] For illustrative purposes only, it can be assumed that the first 
neuron is fired. The firing of the first neuron causes an increase in the voltage of 
a pre-synaptic connection (e.g., a pre-synaptic electrode), as indicated at block 
30 2708, and the an activation of a subsequent or second neuron, as illustrated at 
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bock 2708, which in turn causes a refractory pulse to decrease the voltage of the 
post-synaptic connection (e.g., a post-synaptic electrode), as illustrated at block 
2710. These operations in turn can generally result in an increased voltage 
between pre-synaptic electrodes and post-synaptic electrodes thereof, as 
5 depicted at block 2712. As indicated at block 2712, the processes illustrated 
beginning at block 2704 can be repeated for subsequent electrodes. 

[00270] The result of the operations described at blocks 2704 to 2712 
occurring many times in succession can produce, as illustrated at block 2716, an 
increased frequency of the electric field between the pre-synaptic and post- 
10 synaptic electrodes, thereby causing, as depicted at block 2718, an increase in 
the alignment of nanoparticles (e.g., nanotubes, nanowires, etc.) and a decrease 
in the electrical resistance between electrodes thereof. The process can then 
terminate, as indicated at block 2720 

[00271] One remarkably useful property of a Knowm™ synapse, which 
renders such a device very appropriate for an adaptive neural network, is that the 
frequency or magnitude of the electric field determines the connection strength. 
Thus, the connections that become frequently "activated" become stronger. The 
question of frequency dependence on synapse formation is actually a question of 
frequency dependence on alignment and connection formation, and can be 
viewed from at least two different perspectives. 

[00272] Before further discussion, it will be helpful to make clear some 
terminology that will aid in the descriptions of the device. In the following 
descriptions, an adaptive network is built from one base neuron circuit. Each 
neuron circuit is fundamentally the same, and the network is build by connecting 
25 the base neuron circuits together to form certain topologies that result in desired 
properties, such as maximizing internal feed-back and memory retention. Thus, 
a complete description can almost be made by describing in detail the function 
and circuitry of an individual neuron circuit, and then studying how large numbers 
of the same base neuron will interact with each other. When the term "Neuron" is 
30 used, it refers to the electrical analog of a biological neuron, not a biological 
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neuron. This includes summation properties, in time and space, and output 
properties (e.g., the ability to generate a relatively low-impedance output signal). 

[00273] Biologically, each neuron can receive signals from other neurons 
at its post-synaptic terminals. Likewise, in a Knowm™ physical neural network, 
5 each neuron can receive signals form it post-synaptic electrodes. Biologically, a 
neuron transfers signals to other neurons via its pre-synaptic terminals. 
Likewise, a Knowm™ physical neural network transfers signals via its pre- 
synaptic electrodes. 

[00274] The connections between the pre-synaptic electrodes of one 
10 neuron and the post-synaptic electrodes of another neuron are formed via nano- 
connections, and these connections can be seen as independent from either 
neuron. In other words, the connections do not belong to either neuron, but aid 
in the transfer of signals form one neuron to another. For example, when one 
says "positive activation of a pre-synaptic electrode", one is simply saying that 
15 the pre-synaptic electrode is raised to a positive voltage. With these 
clarifications, we can now proceed. 

[00275] An adaptive network based on nanotechnology fabrication 
techniques can be based on, for example, the use of gold nanowires. Gold 
nanowires are not considered a limiting feature of the present invention, but are 

20 described herein for general illustrative and edification purposes only and also to 
indicate one possible embodiment of the present invention. Gold particles of 
gold ranging in a diameter of approximately 15nm to 30nm can be placed 
between electrodes deposited on a surface. When an alternating voltage is 
applied to the electrodes, thin metallic fibers begin to grow on the electrode edge 

25 facing the gap. 

[00276] The fibers can grow in the direction of the other electrode until the 
gap is bridged, with the wires remaining in contact after the electric field is 
removed. The nanowire growth is caused by particle aggregation at the tip of the 
fibers, thereby extending them toward the opposite electrode. The tip of the 



Attorney Docket No. 1000-1215 
Page 78 of 105 



growing nanowire can create local electric fields of high intensity and gradient, 
giving rise to a dielectrophoretic force, which causes the aggregation. 

[00277] Thus, a first perspective of synapse connection can be seen as a 
bridge-building process, occurring from one electrode to the other. This process 
5 can also be implemented utilizing carbon nanoparticles, such as carbon 
nanotubes and/or carbon nanowires. Because of their exceedingly small size, 
carbon nanotubes present a promising possibility because nanotubes have been 
found to form connections between electrodes. As the frequency increases from 
0 Hz to 10's or even 100's of Megahertz, the standard deviation of angles of 
10 nanotubes from the electric field decreases. Thus, instead of a bridge-building 
process, universal alignment of all nanotubes can be implemented. 

[00278] The nanotubes and/or nanowires and/or other molecular 
conductors (e.g., molecules), however, may join end to end to bridge a gap, 
because many carbon nanotubes may overlap between two or more electrodes 
15 to form long ropes. With this in mind, the space between two electrodes can be 
viewed as a multiplicity of bridges between electrodes (conduits), each separated 
by a characteristic distance that is a result of the local disturbed electric field 
around each "bridge". Likewise, many conduits can bundle together to form a 
rope, bridging the electrodes. 

20 [00279] By configuring nanotechnology-based neural circuitry, as 

described herein, which activates the connections required to be strengthened 
more frequently, while leaving the connections that need to be weakened 
inactivated, an adaptive network can be directly emulated utilizing artificial 
synapses that can both compute and store weight values. All that is required for 

25 such a network is a special type of neuron, modeled very much like that of a 
biological neuron. Each neuron can contain a number of separate functions. By 
connecting enough neurons into a topology that allows internal feedback, a 
modifiable network can be constructed which learns though an adaptive 
feedback process called adaptive integration learning. 
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[00280] As will be discussed later, these neurons can be constructed in 
conjunction with Knowm synapses to form highly interconnected networks, which 
use very little space on a VLSI chip. As one will see, such synapses lead 
themselves to a vertical stacking of planar chips, creating very high-density 
5 neural networks. The primary feature of an adaptive network is that of Hebbian 
learning. If the pre-synaptic neuron fires in close temporal proximity to the post- 
synaptic neuron, then the connections between the two (i.e., the synapse) can be 
strengthened. Similarly, if a synapse remains inactive for a long duration, then 
that connection can be gradually weakened. 

10 [00281] Although this mechanism is by no means proven biologically, it 

provides a possible mechanism of how a cell can strengthen connections in a 
Hebbian manner, and can be considered the basis for an artificial neuron, which 
in turn can be used to build an adaptive network. In any case, some mechanism 
must be in place within a neuron so that its post-synaptic junctions know when 

15 they have contributed to neural activation. In an adaptive network utilizing 
Knowm™ synapses, for example, neurons that achieve a higher rate of activation 
can produce a higher-frequency (and magnitude) electrical field across their 
synaptic connections, thereby strengthening such connections. As one can see, 
the property of Knowm™ synapse strengthening in proportion to an increase in 

20 the frequency of an electric field is seminal to the incorporation of a Knowm™ 
device or component into an adaptive network. 

[00282] Although the connection modification process is the result of an 
overall applied frequency, it can also be seen as a very small incremental change 
for every activation of the synapse. An activation of a synapse can be seen as 

25 an activation of a pre-synaptic neuron at the same time as the activation of the 
post-synaptic neuron. This results in an increase in the electric field, and 
although the connection strengthening process only works with applied 
frequencies, it can be regarded as an incremental change for every activation to 
aid in understanding the behavior of the circuit since it is very difficult to picture 

30 anything more than small time intervals when dealing with large networks 
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operating at high frequencies and summing signals in a temporal manner over 
thousands of synaptic inputs. 



[00283] Signals comprising a voltage from a connection network can be 
summed by a neuron, in a spatial and temporal manner, and compared to the 
5 threshold voltage. One should note that although the voltage from the 
connection network is shown forming a voltage divider with R b , which then in turn 
is summed by the neurons summing circuits, any circuit that accomplishes the 
same take may be used. For example, an amplifying stage may be added, or the 
integrating function may become part of the temporal summing circuit. The 
10 components described herein are meant to outline basic parts for circuit 
operation, but are not intended to limit the scope and type of circuit embodiments 
and implementations thereof. 

[00284] Evaluating an individual adaptive neuron, if the current state of 
activation exceeds the threshold voltage, then a pulse may be emitted. During 

15 this pulse, the RPG, "refractory pulse generator" grounds the post-synaptic 
electrodes. This causes the synapses receiving pre-synaptic activation to 
experience an increase in the local electric field. For example, suppose a pre- 
synaptic neuron just fired, and caused the firing of the neuron. This means that 
the pre-synaptic electrode, which itself is connected to the pre-synaptic neuron 

20 which just fired, is now on the positive swing of the output pulse. 

[00285] The neuron, once fired, can output the same pulse, but the RPG 
will turn it into a negative-going pulse at its post-synaptic electrode. Because the 
firing of the pre-synaptic neuron precedes that of the post-synaptic neuron, the 
respective synapse can see (i.e., experience) an alternating electric field. Thus, 

25 synapses that contribute to the activation of the neuron can receive both an 
increase in the local electric field parallel to the connection direction and, when 
applied many times over, can also experience a higher frequency of activation, 
two parameters that increase the strength of a Knowm™ synapse. For reasons 
that will become clear later, the neuron is allowed to source current only on the 

30 positive portion of the pulse and sink current on the negative portion of the pulse, 
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where as the neuron cannot source or sink current if it is not activated (i.e., 
producing a pulse). 



[00286] One important point to understand before we continue is that each 
neuron operates in a completely asynchronous mode. This results in every 
5 neuron being completely independent from the rest of the network, and 
consequently massively parallel networks can be built that rely on the emergent 
behavior of all the interconnected, independent neurons. General properties of 
the neurons in the network, such as threshold, refractory period and habituation, 
may be controlled externally via an external CPU. Such external inputs may 
10 affect how the network computes, but are not a source of computation in and of 
themselves. 

[00287] Before a neuron can act on information coming from its post- 
synaptic activations, the signals must all be summed. This summation can be 
performed with, for example, a resistor, R b , acting as a voltage divider. 

15 Summation can alternatively be accomplished with an operational amplifier 
circuit, which has the added benefit that parameters can be manipulated 
remotely. A summation circuit can lead to the ability to easily form inhibitory 
connections and even control the activity of excitatory and inhibitory connections 
by adjusting the gain of the excitatory and/or inhibitory amplifiers. In a physical 

20 chip structure, the area taken (on the chip) for the implementation of a 
summation circuit, or the use of a one or more large resistor such as R b , can be a 
deciding factor in what type of circuit is utilized. Many summations circuits exist, 
and it is anticipated that a circuit that offers both external control and low 
component count will be most desirable. 

25 [00288] One of the most important features of an adaptive Knowm™ 

neuron is an integrator. The integrator can sum the signals in time, so that a 
signal received from one synapse in one instant can be added to a signal from 
another synapse a short time later. The integrator one uses in an adaptive 
neuron has a large influence on the behavior of the network. A good analogy is 

30 to consider a barrel with a small hole in the bottom, and a trigger that opens a 
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large valve at the bottom of the barrel when the level of incoming water reaches 
a certain point or threshold. Thus, we can picture pulses of water filling the 
barrel, from various sources, and a constant leakage of water out of the barrel 
due to the small hole. If the rate of water into the barrel is greater than the 
5 leakage due to the hole, the water level will rise until it hits a point where the 
valve is triggered and the water is rapidly flushed from the barrel. 

[00289] In this analogy, the integrator can be seen as the barrel (which 
stores inputs from past time periods) and the leakage hole, which serves to keep 
the integrator from accumulating water and firing over long lengths of time when 
10 little activation is present. Both of these parameters can be adjusted. The 
integrator can be built in a number of different ways, and it is the intent of this 
patent to cover all possible cases, with no preference toward any particular 
circuit. It is anticipated, for example, that the capacitance of the post-synaptic 
electrodes could be used as a stage in the construction of an integrator. 

15 [00290] Some integrating circuits have been found to be more stable than 

others. The exact details of the integration circuit are not important for the 
description of the device described herein, in accordance with one possible 
embodiment of the present invention, because it is the intent of such an 
embodiment to cover generally all possible integrator circuits. It is believed that 

20 the type of integrator utilized can have a large effect on the performance of a 
network. Bi-stable integrators, for example, can be adapted for use in 
accordance with an embodiment herein. The bi-stable property can be emulated 
in the electronic circuitry of the integrator and incorporated into an adaptive 
neuron. 

25 [00291] The next important sub-system of an adaptive neuron is a 

threshold circuit. This can be accomplished via a number of ways, but a 
comparator (i.e. Op-amp) provides the simplest example. If the output from the 
integrator circuit reaches a voltage equal or greater than a threshold voltage, the 
threshold circuit outputs a signal, which we will assume to be high but could also 

30 be low. The threshold circuit, in combination with the integrator and summation 
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circuit, performs the temporal and spatial summations necessary for an adaptive 
neuron. 

[00292] The next sub circuit is the pulse-generator. Because Knowm™ 
networks respond to applied frequencies of electrical fields, it is necessary to 
5 encode outputs via pulses. This idea fits nicely with the biological analogy where 
signals are transmitted as a series of action potentials traveling through axons 
and dendrites. The input to the pulse generator is the output from the threshold 
circuit. The output from the pulse generator is, as can be expected, a pulse. The 
width of the pulse can be determined by the designer of the circuit, or controlled 
10 externally. 

[00293] For reasons that will become clear in the context of an adaptive 
network, the output of the RPG can be a high (+Vcc) pulse followed immediately 
by a low (-Vcc or ground) pulse if the neuron output pulse is a low pulse (-Vcc or 
ground) followed by a high (+Vcc) pulse. Likewise, the output of the RPG can be 
15 a low pulse (-Vcc or ground) followed by a high (+Vcc) pulse if the neuron output 
pulse is a high (+Vcc) pulse followed immediately by a low (-Vcc or ground) 
pulse. 

[00294] The output of the adaptive neuron, as far as any post-synaptic 
neurons are concerned, is generally (although not necessarily) that of the pulse 

20 generator, with one small caveat. To implement the form of Hebbian learning via 
a refractory pulse, as mentioned previously, the adaptive neuron should be 
allowed to strengthen those post-synaptic synapses that are activated while the 
neuron is also activated. In other words, pre-synaptic neuron firings that are 
highly correlated with post-synaptic neuron firings should be strengthened. To 

25 accomplish this, a refractory pulse generator can be introduced. The refractory 
pulse generator takes a positive input (+Vcc from the pulse generator output) and 
produces a negative (-Vcc or ground) pulse at the post-synaptic electrodes. 
Following the negative pulse, the refractory pulse generator creates a positive 
(+Vcc) pulse, which can serve to positively activate the post-synaptic connections 

30 immediately after the negative activation. 
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[00295] Alternately, the refractory pulse generator can produce a positive 
pulse followed by a negative pulse. If the output of the pulse generator is a 
positive pulse followed by a negative pulse, then the output of the refractory 
pulse generator can be a negative pulse followed by a positive pulse. The width 
5 of this positive pulse can be adjusted, but it can be assumed to be the same as 
the negative pulse to aid in the description. One can also think of the refractory 
pulse generator as an inverter of the pulse generator, the output of which 
projects to the post-synaptic electrode. 

[00296] Connected to the pulse generator and refractory pulse generator 
10 and neuron output are two important sub-circuits, a Selective Current Sink and a 
Selective Current Source, which serve an important, although not immediately 
obvious purpose. When the pulse generator outputs a positive pulse, the 
Selective Current Source allows the neuron to source current. When the 
refractory pulse is negative, the selective current sink allows the neuron to sink 
15 current. If no pulse is present, i.e., the pulse generator is outputting zero, then 
the current sink and current source does not sink or source current, but leaves 
the pre-synaptic electrodes floating. 

[00297] The importance of the Selective Current Sink will become clear 
when one considers the group behavior of many neurons in a perpendicular 

20 array structure, all highly interconnected. Such restrictions on adaptive neurons 
can restrict current flow to predominantly the pre-to-post synaptic electrode 
direction, which can also keep unwanted current flows and voltage drops 
occurring from activated post-synaptic electrodes to inactivated post-synaptic 
electrodes and also from activated pre-synaptic electrodes to inactivated pre- 

25 synaptic electrodes. 

[00298] In biological neural networks, habituation of the individual neurons 
plays a large role in global network function. As such, it can be appreciated that 
such an electrical analog could provide useful computational properties. 
Biologically, a neuron needs to consume chemical resources to provide the 
30 energy to fire. When the resources run low, and the by-products overwhelm the 
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cell, the neurons firing rate begins to slow down. Likewise, when the neuron has 
not fired for awhile, the chemical resources needed for energy production begin 
to stockpile, which causes them to fire at heightened frequencies in comparison 
to neurons that have fired more frequently and not built up chemical reserves. 
5 The electrical analog could be provided in many ways, such as making the 
threshold of the neuron a function of the neurons past firing history. This can be 
accomplished with digital and/or analog circuitry, as long as the synaptic 
electrodes receive the proper pulse. 

[00299] On a superficial level, it ca be appreciated how such a Hebbian 
10 learning circuit generally functions. When pre-synaptic neurons activate the 
adaptive neuron, the refractory pulse generator grounds (i.e., or lowers to -Vcc) 
all of the post-synaptic electrodes, which can cause an increased electrical field 
across all connections with high pre-synaptic electrodes. Immediately after the 
negative activation, the refractory pulse generator positively activates the post- 
15 synaptic electrodes. Those pre-synaptic neurons just activated can sink a 
current for a brief time, because of the selective current sink at their output. Thus, 
the connections participating in the activation of the neuron will see a full-wave 
alternating electrical field of increased magnitude. When this process is 
repeated, selective connections (i.e., those with a temporal correlation in firing) 
20 will generally experience an increase in the strength and frequency of the electric 
filed, and consequently become stronger. 

[00300] Synaptic connections that activate just before a neuron's 
activation becomes stronger can experience an increased alternating electric 
field parallel to connection direction. Additionally, synapses that fire after the 

25 neuron activates become weaker (via columbic repulsion), and connections that 
could result in more efficient signal transduction synapses that fire and do not 
activate the neuron become slightly stronger (i.e., experience a half-magnitude 
alternating electrical field). This last form of connection modification provides a 
form of connection exploration within the circuit. Without a form of non-Hebbian 

30 connection formation, potentially useful connections would never form. In other 
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words, for Hebbian learning to take place, the connections should already exist. 
Hebbian learning only "picks out" those connections that turn out to be useful, 
and destroys those that cause undesired outputs. 

[00301] The pulse emitted from the neuron can also take on a variety of 
5 other forms, such as, for example, a sinusoidal pulse, triangular pulse, etc. A 
general concept of how the electric fields at a synapse functions can be obtained 
if one assumes that the frequencies of pre- and post-synaptic activations are not 
the same, and considers the beat-frequencies of the input and output wave-forms 
present at the input and output electrodes. In this case, the gradient of the 
10 voltages at the pre- and post-synaptic electrodes can be approximately 
equivalent to the electric field. Although the connections themselves play an 
important role in the local electric field, it can be assumed that the cross-sectional 
area of the pre- and post-synaptic electrodes is large compared to the size of 
nano-connections, so that this problem is minimized. 

15 [00302] As indicated in the background section of this disclosure, 

researchers, researchers in the neuro-biological fields have been challenged with 
a need to develop a computationally efficient algorithm that can emulate a 
biologically realistic neural network. Specifically, researchers have attempted to 
develop a method, which would allow the efficient calculation of Spike-Timing 

20 Dependent-Plasticity (STDP), while also permitting fully interconnected networks. 
In STDP, timing between pre- and post-synaptic events can cause a net 
potentiation (LTP) or a net depression (LTD) of synapses. FIGS. 28 and 29 
illustrate respective graphs 2800 and 2900 of varying STDP models. 

[00303] Many biological studies have found a relationship between "T", the 
25 inter-spike interval and Ag/g, the fractional change in conductance in the 
synapses, which would indicate LTP or LTD. FIG. 28 illustrates a graph 2800 
indicative of this relationship. Graph 2800 indicates that if the pre-synaptic pulse 
arrives after the post-synaptic pulse, the synapses can be depressed (weakened) 
in an exponential manner. If the post-synaptic pulse arrives after the pre- 
30 synaptic pulse, then the synapse will potentiate, again with an exponential 
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dependence on "7". Other biologically realistic models have predicted a 

i 

dependence such as that provided in graph 2900 of FIG. 29. Thus, graph 2900 
is symmetrical and strikingly different from that of graph 2800 of FIG. 28. 

[00304] To achieve a computationally efficient STDP algorithm, every 
5 neuron outputs a characteristic pulse via its axon, after attaining a threshold 
value. Simultaneously, the neuron can emit a characteristic pulse via its 
dendrites. FIG. 30 illustrates a schematic diagram of a neuron 3000, including a 
dendritic pulse and an axonal pulse thereof. Such pulses accomplish two goals. 
First, the axonal pulses can cause the excitation or inhibition of other neurons. 
10 Second, interactions between dendritic and axonal pulses can determine updates 
to synapses. Each neuron can be designated as either excitatory or inhibitory, 
and the pulses can be designed so that the interaction of a dendritic pulse and an 
axonal pulse results in either potentiation or depression on the synapse, 
depending on the timing between the pulses. 

15 [00305] An update to the synapse can be a function of the pre- and post- 

synaptic pulses. In general, the following rule can be applied: cj t+1 =u) t - 
(pre)(post), where (pre)(post) represents the product of the pre- and post- 
synaptic activation, as provided by a pulse. Consider the pulses 3100 depicted 
in FIG. 31. Two types of pulses are illustrated in FIG. 31, including an excitatory 

20 axonal pulse and an excitatory dendritic pulse. 

[00306] Also, consider the configuration of FIG. 32, which illustrates a 
schematic diagram of two neurons 3200, identified as neuron "A" and neuron "B". 
Neuron "A" can emit a pulse down its axon, and neuron "B", when activated, can 
emit a pulse up its dendrite. When neurons "A" and "B" initiate pulses after 

25 activation, such as, for example, pulses depicted in FIG. 31, a configuration of 
pulses, such as pulses 3300 depicted in FIG. 33, can be achieved. In FIG. 33, T 
is generally a measure between the post-synaptic pulse (dendritic pulse) initiation 
and the pre-synaptic pulse (axonal pulse). Note that in FIG. 33, the dashed lines 
generally represent a post-synaptic pulse (dendritic pulse), while the solid lines 

30 generally represent the pre-synaptic pulse (axonal pulse). 
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[00307] A similar but varying format can be followed in FIG. 35. If an 
update is provided at every time-step, and there are at least four time steps 
available per pulse, such a measurement is generally illustrated in graph 3400 of 
FIG. 34, which is a good fit to realistic biological models incorporating bio- 
5 chemical processes. FIG. 35 represents an alternative set of pulses 3500. As 
indicated at block 3502, other learning rules can be obtained by modifying the 
dendritic or axonal pulses. When the pulses 3502 are plotted, a graph 3600 as 
indicated in FIG. 36 can be generated, which is an approximation of another 
learning rule. 

10 [00308] Thus, by varying the pulse shapes, many different learning rules 

can be implemented. By changing the pulse patterns, post synaptic, pre-synaptic 
or both, different learning rules can be obtained. For example, by using skewed 
gaussian pulses in place of square pulses in the previous example, one can 
obtain an exponential dependence on T\ instead of a linear. The examples 

15 provided herein with respect to the use of pulse generation and pulse shapes are 
merely examples of embodiments in which the present invention can be 
implemented are not considered limiting features of the present invention. 

[00309] FIG. 37 illustrates a high-level block diagram illustrating a system 
3700 comprising a network of nanoconnections 3708 formed between one or 

20 more respective input and output electrodes 3702 and 3704, in accordance with 
an alternative embodiment of the present invention. Nanoconnections 3708 are 
located within a connection gap 3710, which is illustrated generally by a dashed 
line in FIG. 37. Nanoconnections 3708 generally are formed as a plurality of 
interconnected nanoconnections. As indicated earlier, connection gap 3710 can 

25 be filled with a solution (e.g., a liquid, gel, etc.). 

[00310] An individual nanoconnection may constitute a nanoconductor 
such as, for example, a nanowire(s), a nanotube(s), nanoparticles(s), or any 
other molecular structures (e.g., molecules). Nanoconnections 3708 can also be 
constituted as a plurality of interconnected nanotubes and/or a plurality of 
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interconnected nanowires. Similarly, nanoconnections 3708 can be formed from 
a plurality of interconnected nanoparticles (i.e. molecules). 



[00311] A major problem with emulating a biologically realistic neural 
network models, with pulse-coded outputs and modifiable synapses, concerns 
5 the extremely computationally expensive nature of the calculations. Whereas 
every synapse in a biological network can modify itself individually according to 
simple rules such as STDP, any calculation must take into account the state of 
the entire system, in essence storing the state in memory and constantly 
updating it. Because of the largely serial nature of both modern computers and 
10 memory, this task can be extremely inefficient, resulting in simulations that take 
many hours, days, or even weeks to achieve only a few seconds of simulated 
"real time". In addition to the difficulties involved in simulation activities, 
knowledge of the general structure of biologically realistic networks is also 
limited. 

15 [00312] Although synaptic plasticity rules may create a network that can 

learn and adapt to its environment, a major step in creating a truly useful network 
is determining a general structure that allows plasticity to further "tune" the circuit. 
Such a general structure is, for example, coded in the genome of every living 
organism. The more simple the organism, the more "hard-wired" the network. It 

20 is anticipated that many useful networks will be developed for relatively simple 
tasks that are currently being accomplished with lower-level organisms such as 
insects, reptiles and birds. 

[00313] Evolution has determined general connection patterns over 
hundreds of millions of years, and encoded this in DNA. Unfortunately, it is not a 

25 simple matter of analyzing DNA to determining a general connection pattern. 
Nor can a brain be efficiently dissected to determine the connection patterns, 
simply because the fundamental synaptic plasticity rules may differ slightly, and 
with a different learning rule comes different connection patterns. In designing a 
pulsed network with modifiable connections, it is very important to allow every 

30 possible connection, but it is not desirable that every connection be "on". For 
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example, the human brain possesses about 100 billion neurons, but each neuron 
is connected to only 10,000 others. 

[00314] Knowing which connections should not be connected is a problem 
that only evolution can solve. This would require the initialization of certain 
5 synapses (i.e., turning some synapse on, while leaving the majority of the others 
off) and then testing the network by subjecting it to stimulus characteristic to the 
environment in which it will be used. If the network performs the job well, those 
particular connections can be recorded, and then production may begin only for 
those connections that are allowed to form. It is very likely, however, that the 
10 synapses that are initialized on the first trial will not be adequate. If not, some 
connections must be turned "on," "off," or some combination thereof, perhaps in a 
random manner that mimics evolution, or perhaps in a systematic manner. 

[00315] The new pattern of connections must be subjected to its 
environment, and evaluated as to its ability to perform the desired task. It can be 
15 appreciated that a standard computer processor cannot achieve the speed 
necessary to simulate every generation, because many generations will most 
likely need to be evaluated, and every generation must have a chance to adapt 
to its environment via STDP rules. The time required to evaluate one generation 
via standard computation methods would most likely take many weeks. 

[00316] In essence, it is necessary to recreate millions of years of 
evolution and many thousands of generations. To accomplish this, a faster 
version of the environment should be provided, and the network must adapt at an 
appropriately speed-up rate. One can accomplish this feat with a Knowm™ 
adaptive network, but synapses musts be turned "on" and "off' for initialization at 
the beginning of every generation. 

[00317] As indicated earlier, computer simulations of neural networks are 
too complicated and slow for efficient and useful applications. Such serial based 
algorithms must initialize connections and then run learning routines and then re- 
initialize, run again, and so forth. Certain connections are desired for 



20 



25 



Attorney Docket No. 1000-1215 
Page 91 of 105 



initialization, while others simply must be turned off. System 3700 therefore 
presents a solution to this problem. A gate 3706 can be located adjacent to the 
connection gap 3710, electrically insulated, and thus provide an electric field for 
nanoconnections 3708, which can be, for example, semi-conducting nanotubes, 
nanowires, nanoparticle and/or other molecular semi-conducting structures. 

[00318] Gate 3706 can be formed from materials, such as, for example, 
aluminum, gold, and the like. Note that the gate 3706 can be insulated from the 
nanoconnections and/or other molecular semi-conducting structures of the 
connection gap 3710 by a material, such as, for example, silicon dioxide or other 
types of oxide insulators. Recall that a connection network of interconnected 
molecular conductors or nanoconductors may be formed within connection gap 
3719. Although silicon dioxide is shown utilized in the illustrated examples of 
systems 3700 and 3800, it can be appreciated that other types of insulating 
materials may also be utilized in place of silicon dioxide and that the use of 
silicon dioxide is not a limiting feature of the embodiments disclosed herein. 

[00319] Other types of insulating material that could be utilized in place of 
silicon dioxide or which could potentially complement the use of silicon dioxide as 
an insulator may include materials such as, for example, aluminum oxide, 
hafnium oxide, zirconium oxide, yttrium oxide and/or silicon dioxide mixed with 
20 transition or rare-earth metals such as zirconium and/or lanthanum. The use of 
silicon dioxide as an insulating material in FIGS. 37 and 38 herein is therefore 
presented for illustrative and exemplary purposes only. 

[00320] Thus, system 3700 is comprised of semi-conducting 
nanoconnections 3708, rather than simply conducting nanoconnections and/or 
25 other conducting molecular structures. The semi-conducting nanoconnections 
3708 can be formed from material such as, for example, carbon, silicon, indium 
phosphide, and so forth. An AC field can be formed across the connection gap 
3710, and made to vary, thereby strengthening or weakening nanoconnections 
and accomplishing STDP rules. 
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[00321] Such a structure can be viewed as analogous to a field-effect 
transistor, in which the source and drain electrodes correspond to the pre- and 
post-synaptic electrodes, and which the gate effectively controls if the connection 
is on or off. What is different about this structure when compared to prior 
5 literature is that the resistance between the source and drain can be modified by 
aligning the nanoparticles with an alternating electric field across the source and 
drain electrodes, thereby implementing the functionality of a synapse with the 
added ability that the synapse can be turned "on" or "off' with the gate voltage. 

[00322] FIG. 38 illustrates a high-level block diagram illustrating a system 
10 3800 comprising a network of nanoconnections 3708 formed between one or 
more respective input and output electrodes 3702 and 3704, in accordance with 
an alternative embodiment of the present invention. Note that in FIGS. 37 and 
38, identical features or elements are indicated by identical reference numerals. 
Thus, system 3800 of FIG. 38 additionally includes logic circuitry 3718, which can 
15 be connected to gate 3706. Such logic circuitry 3718 can include devices such 
as NAND, NOR, OR, AND logic circuitry and so forth. 

[00323] Logic circuitry 3718 can thus include additionally circuitry such as 
transistors, resistors, capacitors, and the like. The use of gate 3706 in 
association with nanoconnections 3708 permits the connections 3708 and/or 

20 connection network thereof to function as a transistor or a group of transistors for 
determining which individual synapses or groups thereof are to be identified as 
activated or deactivated. System 3700 of FIG. 37 and by extension, system 3800 
of FIG. 38, permit individual synapses within the physical neural network thereof 
to be turned "ON" or "OFF". Thus, systems 3700 and 3800 can be utilized in the 

25 context of a developers chip or a training chip for initialization and/or re- 
initialization and training of the physical neural network formed thereof. 

[00324] FIG. 39 illustrates a block diagram of a physical neural network 
3900, which can be implemented in accordance with an alternative embodiment 
of the present invention. Physical neural network 3900 comprises a plurality of 
30 randomly connected physical neurons 3902 which are created utilizing 
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nanotechnology as indicated herein. A plurality in inputs h to l n can be input to 
physical neural network 3900 and a plurality of outputs d to O n can be 
generated from physical neural network 3900. Physical neural network 3900 
generally comprises a Knowm™ enabled neural network. Network 3900 is 
5 implemented as a web or tangle of Knowm™ enabled neurons. Recall that as 
utilized herein, the term "Knowm" generally refers to a physical neural network 
implemented utilizing molecular technology, including nanotechnology. 

[00325] An example of a Knowm™ enabled physical neural network is a 
physical neural network configured utilizing nanotechnology, wherein such a 

10 physical neural network includes one or more neuron-like nodes that sums one 
or more input signals and generates one or more output signals based on a 
threshold associated with one or more of the input signals. Such a Knowm™ 
enabled physical neural network also can include one or more connection 
networks associated with one or more of the neuron-like nodes, wherein each 

15 connection network comprises a plurality of interconnected nanoconnections, 
such that each nanoconnection of the plurality of interconnected 
nanoconnections can be strengthened or weakened according to an application 
(i.e., increase or decrease) of an electric field and/or frequency thereof. 

[00326] The network structure of such a Knowm™ enabled physical neural 
20 network can be formed such that the nanoconnections themselves are located 
within a solvent, thereby absolving a need for separate weight memory storage 
and processing architectures thereof. The nanoconnections can comprise 
molecular structures such as nanotubes, nanowires, nanoparticles and the like. 
The solvent, as indicated herein, can be a dielectric solvent, liquid crystal media, 
25 a gas, a liquid solution, a gel-based solution and the like. 

[00327] Physical neural network 3900 can be implemented as a relatively 
large physical neural network. For example, physical neural network 3900 can 
be implemented with one or more Knowm™ enabled neurons, including dynamic 
synapses thereof, which possess a high probably of transmission and/or a high 
30 plasticity. Physical neural network 3900 also operates based on an appropriate 
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connectivity. Such a connectivity should not be so high as to drive the network 
to chaos, but not so low that there is not sufficient reverberations in network 
3900. Physical neural network 3900 can also be implemented as an adaptive 
neural network. 

[00328] FIG. 39 illustrates a system 3900 of interconnected neural circuitry 
referred to in the art as a Liquid State Machine, which can be adapted for use in 
accordance with an alternative embodiment of the present invention. Physical 
neural network 3900 thus comprises a Knowm™ enabled liquid state machine. 
System 3900 generally describes a neural network learning mechanism which 
can be applied to a physical neural network formed utilizing nanotechnology, as 
described herein. Such a network generally consists of two or more distinct 
neural modules. Inputs are presented to the first module, referred to as a Liquid 
State Machine or LSM. The LSM is generally a randomly connected network of 
neural circuits. Although the connections may be random, this is not always the 
case. Generally, the exact nature of the connections are not as important as the 
statistics of the connection, such as the amount of interconnectivity. However 
such a LSM is connected, its sole purpose is to provide what is referred to in the 
art as an "analog fading memory". In a liquid state machine, memory tends to 
fade, similar to the fading of ripples associated with liquid, such as water, as a 
result of input (e.g., a rock thrown in a pond) to the liquid or water at various 
times and locations thereof. 

[00329] The LSM can store, via patterns of neural activations, its recent 
past history. Other types of neural circuits can be utilized to extract the "state" of 
the LSM. A state-extracting neural circuit can be accomplished by a very simple 
25 learning neuron, such as, for example, a perceptron. Such perceptrons can 
adjust their synaptic weights so as to produce a desired output. Such 
perceptrons can be referred to as a "read-out" neuron. The exact rule that the 
read-out neurons utilize may vary, but in general such read-out neurons can form 
a simple linear mapping between the neural circuits within the LSM and the read- 
30 out neuron output. 
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[00330] FIG. 40 illustrates a block diagram of a system 4000 that includes 
physical neural network 3900 of FIG. 39 in association with a plurality of 
perceptrons Pi to P n , which can be implemented in accordance with an 
alternative embodiment of the present invention. Perceptrons Pi to P n can be 
5 implemented in a manner that permits such perceptrons to make random 
connections into the liquid state machine of physical neural network 3900. The 
perceptrons Pi to P n can be taught to "learn" from the output of the information 
provided by physical neural network 3900 in order to achieve temporal pattern 
recognition. Pi to P n of FIG. 40 generally comprise supervised learning devices, 
10 whereby connection strengths are determined by pre-synaptic activity and/or 
post-synaptic error activity, wherein the error is a function of the actual and 
desired outputs of physical neural network 3900. Perceptrons Pi to P n can be 
implemented utilizing nanotechnology as indicated herein. 

[00331] The general structure of the network depicted in FIG. 40 may be 
15 implemented via a Knowm™ enabled physical neural network. FIG. 41 illustrates 
a schematic diagram of a physical neural network 4100, which can be 
implemented in accordance with an alternative embodiment of the present 
invention. Physical neural network 4100 of FIG. 41 is analogous to physical 
neural network 3900 of FIG. 39. Physical neural network 4100 comprises a liquid 
20 state machine of the type indicated the aforementioned Maas et al reference. 

[00332] A legend 4102 is indicated in FIG. 41, which relates to 
components and elements of physical neural network 4100. A detail portion 
4104 of physical neural network 4100 is shown in detail in FIG. 41 and illustrates 
a dendritic electrode 4108 and an associated axonal electrode 4106. A plurality 

25 of nanoparticles 4110 and 4112 are respectively located between the dendritic 
electrode 4108 and an associated axonal electrode 4106. It can be appreciated 
that the same feature or pattern of detail portion 4104 of FIG. 41 is generally 
repeated throughout physical neural network 4100. Physical neural network 
4100 illustrates an example 24 neuron Knowm™ enabled physical neural 

30 network. It can be appreciated that the number "24" is indicated herein for 
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general illustrative and edification purposes only and that such a number can run 
into the thousands, millions and potentially billions. 

[00333] As depicted in FIG. 41, the LSM may be implemented in Knowm™ 
enabled physical neural network by placing neural circuitry in such a pattern so 
5 as to arrange for near-crossings of pre- and post-synaptic electrodes. An 
example of such a "near crossing" is indicated in FIG. 41 by dendritic electrode 
4108 and axonal electrode 4106. Although FIG. 41 generally illustrates dendritic 
electrode weaving under the axonal electrode at a perpendicular angle, such a 
configuration is not a limitation of the embodiments disclosed herein. Any 
10 arrangements that bring a pre-synaptic electrode close to a post-synaptic 
electrode will suffice. 

[00334] As illustrated by nanoconnections 4110, a Knowm™ enabled 
synapse can be formed between dendritic electrode 4108 and axonal electrode 
4106. Although FIG. 41 illustrates that every connection in network 4100 can be 

15 present, not all connections may be desired. Such a configuration can be 
accomplished via simply by-passing the connection in the fabrication of the 
Knowm™ enabled physical neural network through an avoidance of brining the 
desired axonal and dendritic electrode close together in the solution (e.g., liquid, 
gel, solution, solvent, etc). Alternately, the connection can be turned "off' if semi- 

20 conducting nanoparticles are utilized and applied to a gate voltage as depicted in 
the example embodiments of FIGS. 37 and 38. 

[00335] To retrieve information from a Knowm™ enabled implementation 
of an LSM, the required read-out neuron circuitry should be provided. FIG. 42 
illustrates a schematic diagram of a system 4200 that includes the physical 

25 neural network 4100 of FIG. 41 and perceptrons thereof, which can be 
implemented in accordance with an alternative embodiment of the present 
invention. Note that in FIGS. 41 and 42, similar or identical parts or elements are 
generally indicated by identical reference numerals. For example, the features of 
detail portion 4104 can be located at locations 4208 and 4206 and elsewhere 

30 within system 4200. System 4200 also includes a plurality of perceptrons 4210, 
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4212, 4214, and 4216. An example of one such perception is perceptron 4216, 
which is illustrated located at detail 4202 of FIG. 42. 



[00336] Such circuitry describes a general circuit that compares the 
desired output to the actual output and applied a feed back signal to selectively 
5 strengthen or weak the connections. Such a feed-back signal could comprise, 
for example, a variable voltage or frequency. System 4200 is merely an 
illustrative example of a system that can be implemented in accordance with an 
alternative embodiment of the present invention. The number or perceptrons, 
neurons, and the like, is not considered a limiting feature of the present invention. 
10 System 4200 is described herein generally to indicate the implementation of a 
physical neural network as a liquid state machine in association with perceptrons 
thereof. System 4200 thus comprises a mechanism for implementing dynamic 
fading memory properties. 

[00337] FIG. 43 illustrates a schematic diagram of a perceptron 4300 (i.e., 
15 a linear read out neuron), which can be implemented in accordance with an 
alternative embodiment of the present invention. Perceptron 4300 generally 
includes a plurality of electrodes 4310, 4312, 4314, 4316, and 4318 and an 
output electrode 4320. A plurality of nanoconnections (i.e., nanoparticles, 
nanotubes, nanowires, and the like) are respectively located between electrodes 
20 4310, 4312, 4314, 4316, and 4318 and output electrode 4320. Output electrode 
is coupled to a resistor 4321 and an amplifier 4324 and additionally to a switch 
4328. Resistor 4321 can be connected to a ground 4322. Output from amplifier 
4324 can be connected to an amplifier 4326, which in turn is coupled to switch 
4328. Perceptron 4300 is analogous to perceptrons Pi to P n of FIG. 40 and 
25 perceptrons 4210, 4212, 4214, and 4216 of FIG. 42. In general, perceptron 4300 
comprises a supervised learning apparatus or mechanism whereby connection 
strengths can be determined by pre-synaptic activity and post-synaptic error 
activity. Error activity is a function of actual and desired output. 

[00338] For example, if the desired output depicted in FIG. 43 is "low" and 
30 the actual output is "high", then the connection strength can be decreased from 
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"high input neurons". Similarly, if the desired output is high and the actual output 
is low the connection strength can be increased from "high input neurons". If the 
desired output is high and the actual output is high, then no action is required. 
Similarly, if the desired output is low and the actual output is low, then no action 
5 is required. In order to increase the connection strength, an increased field 
gradient can be applied across the connection gap (i.e., containing nanoparticles 
in solution) or the frequency can be altered across the connections thereof, 
whether "up" or "down". To weaken the connection strength, the connection(s) 
can be charged by raising the pre- and post-synaptic electrodes to a high 

10 voltage. Because the dieltrophoretic force responsible for particle chaining is 
frequency-dependant, the electrical frequency across the pre- and post-synaptic 
electrode can also be altered in order to weaken the connection strength. Thus, 
amplifiers 4326 and 4324 can be regarded as a general mechanism to provide a 
feedback signal, either through a steady-state voltage signal or an increased or 

1 5 decreased frequency. 

[00339] Based on the foregoing, it can be appreciated that embodiments 
disclosed herein illustrate a physical neural network, which can be configured as 
a liquid state machine. Such a physical neural network can be configured from 
molecular connections located within a dielectric solvent between pre-synaptic 

20 and post-synaptic electrodes thereof, such that the molecular connections are 
strengthened or weakened according to an application of an electric field or a 
frequency thereof to provide physical neural network connections thereof. A 
supervised learning mechanism can also be associated with the liquid state 
machine, whereby connections strengths of the molecular connections are 

25 generally determined by pre-synaptic and post-synaptic activity respectively 
associated with the pre-synaptic and post-synaptic electrodes. The liquid state 
machine generally can comprise a dynamic fading memory mechanism. 

[00340] The embodiments and examples set forth herein are presented to 
best explain the present invention and its practical application and to thereby 
30 enable those skilled in the art to make and utilize the invention. Those skilled in 
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the art, however, will recognize that the foregoing description and examples have 
been presented for the purpose of illustration and example only. Other variations 
and modifications of the present invention will be apparent to those of skill in the 
art, and it is the intent of the appended claims that such variations and 
5 modifications be covered. 

[00341] The description as set forth is not intended to be exhaustive or to 
limit the scope of the invention. Many modifications and variations are possible in 
light of the above teaching without departing from the scope of the following 
claims. It is contemplated that the use of the present invention can involve 
10 components having different characteristics. It is intended that the scope of the 
present invention be defined by the claims appended hereto, giving full 
cognizance to equivalents in all respects. 
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